1  ftD-Rl 

UNCLA! 

50  324  HIERARCHICAL  HULT I SENSOR  IHAQE 
HONEVHELL  SVSTENS  AND  RESEARCH 

R  K  AGGARHAL  ET  AL.  AUG  S5  AFOSI 
5SIFIED  F49620-82-C-0124 

JNDERSTANDING(U) 
CENTER  NINNEAPOLI5 
I-TR-83-0801 

F/G 

HN 

2/1 

1/2 

NL 

| 

1 

! 

1 

1 

B 

n 

n 

Vi 

'W 

n 

n 

•pa 

AFOSF  TR- 


<M 

00 


< 

I 

D 

< 


Hierarchical  Multisensor 
Image  Understanding 


Final  Report 


August  1985 


Honeywell 

Systems  and  Research  Center 

2600  Rid g way  Parkway 
Minneapolis,  Minnesota  55413 

B 


DT1C 

elect* 


OCT  1  5  fi8S 

^  A 


0 


Air  Force  Office  of  Scientific  Research 
Air  Force  Systems  Command 
United  States  Air  Force 


i  C  1  } 


■ecuri  r  v  classic  icai  ion  ok  i  his  pai.i  (when  ua  i  ft  in'i  mu) 

REPORT  DOCUMENTATION  PAGE 


READ  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 


.ll.  REPORT  NUMBER 


AFOSR-TR-  8  5-  0  80 


2.  GOVT  ACCESSION  NUMBER  I  3.  RECIPIENT’S  CATALOG  NUMBER 


4.  TITLE  (ANO  SUBTITLE) 


Hierarchical  Multisensor  Image  Understanding 


TVPt  OF  REPOM  'PERIOD  COVf  RED 

Final  Report 

October  1983  -  August  1985 


6.  PERI  ORMINC.  ORG.  REPORT  NUMBER 


?.  AUTMC'R(S) 

Raj  K.  Aggarwal  Youn  Kim  Nelson  Marquina 
Mike  Bazakos  Steve  Mader  Firooz  Sadjadi 
John  Budenske  Alianna  Maren  Rand  Whillock 


9.  PERFORMING  ORGANIZATIONS  NAMt/ADDRESS 

Honeywell  Systems  &  Research 
2600  Ridgway  Parkway 
Minneapolis,  Minnesota  55413 


M  I  <  if'  (.RANT  NtiMHI  li;S| 


F49620-33-C-0134 


10.  PROGRAM  ELEMENT  PROJECT, TASK  AREA 
A  WORK  UNIT  NUMBERS 


14.  MONITORING  A 


£\\oa*=  (\~\ 


12.  REPORT  DATE 

August  1985 


13.  NUMBER  OF  PAGES 


S  (IF  DIFFERENT  FROM  CONT.  OFF. |  115.  SECURITY  CLASSIFICATION  (OF  THIS 

Unclassified 

I  IS..  DECLASSIFICATION  DOWNGRADING  SCHEDULE 


16.  DISTRIBUTION  STATEMENT  (OF  THIS  REPORT) 


Approved  for  public  release;  distribution  unlimited 


17.  DISTRIBUTION  STATEMENT  (OF  THE  ABSTRACT  •  NT  I  RE  f)  IN  Bl  IK  K  20.  If  Olf  f  ERENT  f  ROM  RCPORT| 


19.  KEY  WORDS  (  CONTINUE  ON  REVERSE  SlOE  IF  NEC  T ‘jSARY  AND  IDFNTIFY  BY  BLOCK  NUMBER) 

Image  processing.  Image  understanding.  Artificial  intelligence.  Scene  analysis. 


& 


20.  ABSTRACT  (CONTINUE  ON  R£  VERSL  SlOE  if  NEC  E  *  SAM  V  AND  IDENTIFY  flV  BInCK  NUMHEUi 

This  report  describes  the  research  results  on  Honeywell  s  Hierarchical  Multisensor  Image 
Understanding  program.  Honeywell  is  developing  a  unified  framework  for  the  different  hierar¬ 
chical  levels  of  image  processing  such  as  segmentation,  detection,  classification,  and  iden¬ 
tification  of  outdoor  scenes  and  across  different  sensor  modalities  such  as  millimeter  wave, 
infrared,  and  visible.  Current  activities  on  the  project  are  reviewed  under  the  following 
headings:  (1)  A  Survey  of  Multisource  Information  Fusion  Systems;  (2)  The  Role  of  Structure 
in  Human  and  Machine  Perception;  (3)  A  Knowledge  Based  Image  Segmentation  System;  (4)  The 
Use  of  Optical  Flow  as  a  Oepth  Cue  in  Scene  Analysis  and  (5)  Belief  Maintenance  for  a  Fuzzy 

_ Ci,#*  ♦  Am 


•  an  t  \  *473  hhm.v  t«.»v  i  i  ,  »  >f 


^  X'p(<  .  L-,-y 


>ti\l  Ml-.  I  A.JL  i  .-.'-EN  D-  I  a  (  N  > 


<LMM 


*IV  F03'~s  r  - 
,Vf>T  '  ’';  -  ■  ' 

Th'.--, 


«AT?H 


Chief,  x 


i-i  Di  ,-i  j ic 


Hierarchical  Multi  sensor 


Image  Understanding 


Final  Report 


\cc "  ion  For 


Contract  F49620-83-C-0134 


!  NTIS  CRA&/ 

OTIC  tab 

Unannounced 

Justification 


Honeywell  Systems  and  Research  Center 


By 

Distribution/ 


Minneapolis,  Minnesota  55413 


Availability  Codes 


Djc*  Avaii  and/or 
/f./  Special 


ABSTRACT 


This  report  describes  the  research  results  on  Honeywell's  Hierarchical 
Multisensor  Image  Understanding  program.  Honeywell  is  developing  a  unified 
framework  for  the  different  hierarchical  levels  of  image  processing  such  as 
segmentation,  detection,  classification,  and  identification  of  outdoor  scenes 
and  across  different  sensor  modalities  such  as  millimeter  wave,  infrared,  and 
visible.  Current  activities  on  the  project  are  reviewed  under  the  following 
headings:  (1)  A  Survey  of  Multisource  Information  Fusion  Systems;  (2)  The 
Role  of  Structure  in  Human  and  Machine  Perception;  (3)  A  Knowledge  Based 
Image  Segmentation  System;  (4)  The  Use  of  Optical  Flew  as  a  Depth  Cue  in 
Scene  Analysis  and  (5)  Belief  Maintenance  for  a  Fuzzy  Reasoning  System.  Past 
activities  on  the  project  which  are  reported  in  Annual  Report  covering  period 
October  1983  -  September  1984  include  (a)  Al-based  Generic  Image  Segmentation 
and  Object  Recognition;  (b)  Evidence- Confidence  Para di ©ns  for  Image 


Understanding;  (c)  Hierarchical  Systems  Theory  for  Control  Structures,  and 

* 

(d)  Invariant  Methods  in  Image  Understanding. 
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A  SURVEY  OF  MULTISOURCE  INFORMATION  FUSION  SYSTEMS 


T/eLE  OF  OON TENTS 


1.0  INTRODUCTION 

2.0  SYSTEM  DESIGN 

3.0  EVIDENTIAL  REASONING 

3.1  Introduction  to  Evidential  Reasoning 

3.2  Bayesian  Methods 

3.3  MYCIN 

3 .4  Dempster  Schafer 

3.5  Edge  Merits 

3.6  Endorsements  and  Endorsers 

3.7  Non-Monotonic  Logic 

3.8  Fuzzy  Sets  and  Logic 
4.0  PLANNING  AND  CONTROL 


5.0  SUMMARY 


The  human  brain  Is  able  to  accept  Information  from  multiple  sources  (l.e.  five 
senses)*  fuse  the  Information  Into  a  single  pool  of  knowledge*  reason  across 
the  knowledge*  predict  which  of  the  sources  will  provide  the  most  beneficial 
amount  of  new  Information*  and  direct  resources  to  those  sources  In  order  to 
concentrate  on  processing  new  Information.  In  this  way  the  human  successfully 
Interacts  with  his  environment*  assessing  and  understanding  ongoing  situations. 

Often  this  Information  Is  the  result  of  partial  scanning  of  the  domain* 
resource-limited  processing*  many-to-less  heuristic  beliefs*  probabilistic 
assumptions*  and  dimensional  mapping.  Thus*  human  processing  uses  Information 
which  can  be  Incomplete*  uncertain*  even  Incorrect,  and  even  Inherently 
evidential.  Each  piece  of  Information  thus  has  some  amount  of  belief 
associated  with  It  based  on  Its  source*  condition  of  collection*  metnod  of 
collection*  etc.  In  other  words*  the  belief  Is  based  on  evidence.  Making  an 
Inference  about  the  world  based  on  the  beliefs  of  the  Information  (and 
associated  evidence)  requires  not  only  reasoning  over  the  Information  but 
reasoning  about  the  belief  of  the  Information  and  the  evidence  that  that  belief 
Is  based  on. 

Being  able  to  manage  multiple  sources  of  Information  Is  a  skill  at  which  humans 
must  be  proficient  In  order  to  Interact  with  their  environment.  The  goal  of 
Intelligent  systems  Is  also  Intelligent  environmental  Interaction.  These 
systems  will  need  multiple  sources/sensors  which  can  provide  a  variety  of 
environmental  Information.  Like  their  human  counterparts*  Intelligent  systems 
which  are  to  Interact  with  their  environments  must  be  able  to  fuse* 
evidentially  reason  over*  and  control  the  processing  of  multiple  sources  of 
uncertain,  complete  and  Incorrect  knowledge.  What  follows  Is  a  survey  of  the 
design  of  such  systems.  In  surveying  the  state-of-the-art  In  Multi  source 
Information  Fusion  (MSIF)  the  research  breaks  down  into  three  subtopics:  The 
design  of  the  entire  MSIF  System*  the  work  done  In  Evidential  Reasoning*  and  In 
Planning  and  Control  of  the  multiple  sources. 


2.0  SYSTEM  DESIGN 
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A  Multi  source  Information  Fusion  System  must  perform  a  multitude  of  tasks. 

The  sensors  must  be  controlled;  the  low  level  data  processed  Into  high  level 
knowledge;  uncertainty  factors  assigned  to  data*  knowledge  and  process /system 
assumptions;  Information  must  be  pooled  and  extrapolated;  assumptions  about 
the  data  must  be  made  and  response  to  these  assumptions  must  be  created* 
planned  out  and  executed.  These  tasks  can  be  shown  by  Figure  1.  In 
searching  the  literature  only  the  work  of  Garvey  and  Lowrance  [Garvey  and 
Lowrance*  84;  Garvey  and  F1sch1er*80;  and  others  In  the  System  Design 
references]  has  addressed  the  Issue  of  a  complete  system  design.  Their  work 
emphasized  a  system  work  listed  for  battle  threat  assessment.  In  their 
design  they  use  four  basic  tasks  (see  Figure  2): 

1)  ANTICIPATE:  Using  current  Information*  this  task  attempts  to 
identify  prospective  significant  events.  The  system  Is  looking  for 
"What  will  happen  next"?  The  knowledge  used  Includes  known 
entitles  In  the  environment*  their  capabilities*  entities  often 
associated  with  them*  etc.  This  module  Is  attempting  to 
hypothesize  what  It  Is  missing  or  could  be  about  to  miss  with  its 
sensors.  This  Information  Is  then  passed  on  to  the  next  task. 

2)  PL AN/ ALLOCATE:  Given  a  list  of  what  coulo  be  happening  In  the 
environment*  this  module  decides  what  Is  Important  to  sense  and  how 
It  will  attempt  to  sense  It.  A  'plan'  Is  created  and  passed  on. 

3)  CONTROL:  From  the  plan*  this  module  guides  and  manages  the 
sensors*  parameters*  and  data  operators.  The  data  collected  are 
passed  onto  the  next  phase. 

4)  INTERPRET:  The  collected  data  are  added  to  the  model  of  current 
situation.  The  new  Information  Is  Inferenced  over  and  the  updated 
world  model  Is  passed  on  to  the  ANTICIPATE  task. 


■r. 

j*,' 


lifc 


2 


Unfortunately  all  of  Garvey's  work  past  thfs  has  been  on  the  INTERPRETATION 
task  where  he  uses  the  Dempster-Schafer  rules  (covered  later)  to  perform 
evidential  reasoning  over  a  hierarchical  world  model  of  threat  behavior. 
Garvey  assumes  that  data  will  be  In  symbolic  form  and  thus  his  knowledge 
representation  and  fusion  mechanisms  do  not  consider  any  non-symbol  1c  levels 
of  sensory  data  (1e.  Image  operator  results).  Possibly  these  non-symbollc 
levels  of  sensory  data  are  unusable  until  they  are  transformed  Into  symbolic 
data.  Since  Garvey  has  concentrated  on  the  INTERPRETATION  task  of  his 
design#  the  details  of  the  PLANNING/ ALLOCATE  and  CONTROL  tasks  are 
unspecified.  Much  more  work  needs  to  be  done  before  Garvey's  system  can 
actually  be  called  complete.  Though  Garvey's  work  appears  to  be  the  only 
published  research  concerning  a  complete  system  design#  many  others  have  been 
researching  the  evidential  reasoning  problem.  There  appears  to  be  a  large 
void  of  research  In  the  'complete'  system  design  and  development  aspects  of 
Multi  source  Information  Fusion*  and  though  the  research  on  evidential 
reasoning  Is  necessary#  Its  development  must  take  Into  consideration  the 
Interaction  with  the  remainder  of  the  system.  This  can  only  be  done  after 
the  design  of  other  components  has  begun. 
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Figure  2:  The  Basic  Loop 


Evidential  reasoning  has  been  the  main  focus  of  Multi  source  Information 
Fusion  systems  research*  because  it  Is  a  defined  subproblem  of  MSIF  systems* 
and  because  many  feel  that  It  Is  the  heart  of  the  MSIF  system.  Solving  this 
problem  will  allow  partial  Implementations  of  systems  and  provide  a  starting 
point  for  Incremental  development.  An  evidential  reasoning  system  first 
needs  an  Internal  model  of  the  world  on  which  to  reason*  deduce  and 
Inference.  Established  world  model  designs  (Rich  83)  and  their  Inference ng 
mechanisms  are  too  rigid  and  Inflexible  to  truly  reflect  the  real  world,  and 
though  current  research  efforts  are  expanding  these  designs  to  represent  the 
real  world*  much  of  this  research  has  only  been  at  the  symbolic  level.  The 
Issue  of  how  the  non-symbol  1c  levels  of  data  are  affected  by  uncertainty* 
Incompleteness*  and  Inconsistency,  and  whether  this  effect  will  affect  the 
remaining  mechanism*  has  yet  to  be  addressed.  (This  Issue  Is  more  accurately 
called  the  Data  Fusion  problem  but  we  will  use  the  terms  Interchangeably). 

In  order  to  reflect  the  real  world*  data  fusion  systems  must  be  able  to 
represent  the  wide  range  of  knowledge  found  throughout  the  real  world.  This 
knowledge  can  be  declarative  or  procedural*  and  due  to  the  conditions  under 
which  It  was  created.  It  will  have  an  uncertainty  associated  with  It. 
Uncertainty  of  the  knowledge  must  be  represented*  even  to  the  point  of  being 
Ignorant  of  the  knowledge.  The  system  can  be  Ignorant  of  a  certain  fact*  and 
must  be  able  to  represent  and  work  around  this  Ignorance  of  Information 
(however,  knowing  Ignorance  of  a  fact  reveals  a  lack  of  total  Ignorance). 
Finally,  when  quantities  of  Information  are  combined  Into  knowledge  bases* 
consistency  becomes  an  Issue.  It  is  important  also  that  the  facts  in  the 
knowledge  base  do  not  conflict.  Methods  such  as  belief  and  truth  maintenance 
systems  do  this  kind  of  consistency  checking. 


The  most  Important  function  of  a  data  fusion  system  Is  to  derive  new 
Information  from  known  Information.  There  are  two  ways  to  derive  new 
Information; 


(1)  Extrapolation  uses  procedural  rules  to  derive  new  Information  from  a 
single  piece  of  evidence*  and 

(2)  Pooling  combines  multiple  pieces  of  evidence. 

When  pooling  evidence  to  form  new  evidence  there  can  be  a  problem  with 
Interdependencies  of  different  pieces  of  evidence.  For  example  in  a  medical 
diagnosis  problem*  two  symptoms  might  occur  pointing  to  the  same  disease. 
However,  this  does  not  always  mean  a  greater  chance  of  the  disease  than  if 
only  one  of  the  symptoms  had  been  observed.  It  Is  possible  that  the  two 
symptoms  are  dependent  and  always  occur  together.  Different  data  fusion 
systems  deal  with  this  dependency  problem  In  different  ways.  Some  even 
Ignore  It. 

In  the  following  sections  three  different  approaches  to  data  fusion  are 
analyzed  and  two  new  theories,  each  addressing  a  slightly  different  problem 
In  evidential  reasoning  are  explained. 


3.2  Bflxeslfln  Methods 

Bayesian  probabilities  as  used  In  Prospector  by  Duda  et  al  (Duda*  Hart  and 
Nilsson  1976)  assigns  strict  probabilities  for  each  piece  of  knowledge  in  the 
system.  The  probability  of  an  event  H  Is  P(H),  where  P(H)  Is  a  real  value 
between  0  and  1.  These  probabilities  must  follow  the  statistical  law  that 
says  the  sum  of  all  of  the  probabilities  for  all  possible  outcomes  of  a  given 
event  must  be  1. 

The  combination  of  the  associated  probabilities  when  facts  are  combined  Is 
handled  by  Bayes  theorem.  Bayes  theorem  says  that  the  probability  of  an 
event  H  based  on  some  observed  evidence  E  Is  given  by: 
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Likewise* 


P(HIE) 


P(HIE) 


P(£1H)  *  P<H) 

- P7E7 - 


_ P(E IH)  * •*  P(H) 

K 

21P1EIH.)  *  P(H.) 
1=1  1  1 


where: 

P(ElHf)  =  the  probability  of  the  evidence  E  given  Hj. 

P(H)  =  the  aprlorl  probability  of  the  event  H. 

K  =  the  number  of  different  possible  outcomes  for  H. 

There  are  some  problems  with  using  Bayes  theorem  for  real  world  applications 
however.  It  requires  knowledge  of  all  of  the  conditional  probabilities*  and 
this  Information  Is  often  not  available.  Bayes  theorem  has  limited 
application  In  a  world  where  we  do  not  have  access  to  the  probabilities  for 
all  of  the  events  we  are  concerned  with. 

The  Bayesian  method  also  has  problems  with  dependence  of  certain  facts.  By 
definition  Bayes  theorem  requires  all  evidences  to  be  Independent.  This  Is 
often  not  true  In  the  real  world. 

Despite  their  shortfalls*  Bayesian  methods  can  be  useful  In  limited  problem 
domains  where  much  Information  about  the  probabilities  of  events  are  known. 
Mineral  exploration,  which  Is  the  domain  for  Prospector*  Is  one  such  area. 

Charnlac  (Charnlac*  1983)  points  out  that  for  many  expert  systems 
applications  such  as  medical  diagnosis*  Bayesian  methods  can  be  used  witnout 
the  Independence  assumption.  He  shows  that  the  Interdependence  of  symptoms 
will  affect  the  probabilities  for  all  diseases  equally  and  will  therefore  not 
change  the  relative  rankings  of  the  diseases.  He  goes  on  to  state  that  other 
dependency  problems  can  be  addressed  by  combining  the  dependent  evidences 
Into  single  states. 

There  are  many  systems  that  use  derivatives  of  Bayesian  for  data  fusion. 
References  to  many  of  these  are  listed  In  the  bibliography. 
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In  MYCIN*  Shortllffe  (Shortllffe,  1975)  attempted  to  overcome  the 
shortcomings  of  Bayesian  approaches  while  retaining  the  advantages*  by  using 
a  system  that  was  an  approximation  to  conditional  probabilities.  Each  piece 
of  Information  In  MYCIN  has  associated  with  It  a  value  between  0  and  1  which 
Is  Its  belief  value:P(H).  The  disbelief  value  Is  l-P(H).  Any  assertion 
about  this  fact  has  two  measures  associated  with  It.  A  measure  of  belief 
(MB)  Is  the  measure  of  the  decrease  In  disbelief  of  H  as  a  result  of  a  piece 
of  evidence  and  a  measure  of  disbelief  (MD)  Is  a  measure  of  the  decrease  In 
belief  of  H  as  a  result  of  the  evidence.  Ml  Is  defined  by: 

| L  :  If  P(H)  «  I 

KB  =  7 


\  MAXCP(HIE),  P(H)]  -  P(H) 

- T-PTh7 - 


:  otherwise 


and  MO  Is  defined  by: 

1 

/ 

MD  * 


:  if  P(H) 


MINCP(HIE),  P(H)]  -  P£H7 


:  otherwise 


In  addition*  a  certainty  factor  Is  computed  which  Is  the  measure  of  belief 
minus  the  measure  of  disbelief. 


The  above  formulas  handle  extrapol atl on  where  a  single  piece  of  evidence 
leads  to  a  conclusion.  To  pool  multiple  pieces  of  evidence  MYCIN  uses  the 
following  formulas: 


reCHISj+Sj] 


:  If  MDCHlSj+Sj]  *  1 


MB[H |S1 3  +  *BCH|S23  *  (l-MDCHISj^)  :  otherwise 


and* 


MOCHISj+Sj] 


:  If  NBCHIS^Sj]  *  I 


M0CH1S13  +  MDCH1S23  *  C1-MDCHIS13)  :  otherwise 
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The  reasoning  system  used  In  MYCIN  allows  uncertainty  and  allows  us  to 
represent  the  lack  of  evidence  for  a  certain  conclusion.  For  example*  If  the 
evidence  does  nothing  to  confirm  a  hypothesis  then  W3*0.  If  a  piece  of 
evidence  does  nothing  to  disprove  a  conclusion  then  MD®0.  This  method  of 
reasoning  does*  however*  still  have  some  shortcomings.  We  still  have 
dependency  problems  just  as  we  did  with  the  Bayesian  methods  but  now  we  also 
have  an  added  disadvantage  In  that  the  measures  of  belief  and  disbelief  are 
not  true  probabilities  and  therefore  cannot  be  expected  to  follow  the  laws  of 
probabilities.  These  values  are  arbitrary*  and  at  some  level  are  assigned  by 
a  human.  This  poses  a  problem  when  Information  from  different  humans  Is 
used  In  the  same  system.  Invariably  they  will  use  different  scales  In 
assigning  values. 

3.4  Dempster- Schafer 

The  Dempster- Schafer  theory  (Barnett*  1981)  uses  the  same  Interval  concepts 
as  MYCIN  but  extends  them  so  that  the  probabilities  are  represented  by  mass 
distribution  functions.  This  theory  provides  rules  for  assigning  and 
manipulating  these  distributions.  The  Intervals  are  defined  as  follows: 

CS(A),  P(A)] 

where:  S(A)  Is  the  degree  of  support* 

P(A)  Is  the  degree 'of  plausibility 

or  the  degree  of  failure  to  refute* 

P(A)-S(A)  Is  the  degree  of  Ignorance. 

These  values  are  computed  using  a  mass  distribution  function  which 
distributes  a  belief  value  over  the  entire  range  of  possible  hypotheses.  The 
value  for  S(H)  or  the  degree  of  support  for  a  given  hypothesis  H  Is  equal  to 
the  mass  distribution  function  summed  over  all  the  hypotheses  that  Imply  H: 

S(H)  ■  2  M( E -j ) 
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The  value  for  P(H)  or  the  plausibility  of  H  Is  the  sum  of  all  of  the 
hypotheses  that  do  not  Imply  not  (H): 


P(H)  «  ^MtEj)  »  1  -  $(H) 
E^H  <>  0 


To  pool  multiple  pieces  of  evidence  to  support  one  hypothesis  we  use 
Dempster's  rule  for  combination: 


M<H)  ”  I-R  M11E|)  «  M2<6  ) 

E/'EJ  *  H 


where 


K  *  W^E,)  *  H2(Ej) 


V‘ej  * 0 


The  Dempster-Schafer  theorem/  because  It  follows  the  law  of  probability* 
gives  us  many  nice  properties.  The  combination  rule  is  commutative  and 
associative  so  evidence  can  be  combined  In  any  order  or  grouping.  Also*  when 
the  probabilities  are  known  exactly  (when  S(H)  «  P( H) )  the  Dempster-Schafer 
law  reduces  to  Bayes  theorem. 


The  Dempster-Schafer  theorem  fills  many  of  the  needs  for  an  Information 
combination  system  and  Is  widely  used  today.  References  to  many  systems  that 
use  the  Dempster-Schafer  theorem  are  listed  In  the  bibliography. 


3.5  Edoe  Merits 


Slagle  (1984)  In  his  battle  system  proposes  an  extension  to  the  Bayesian  and 
MYCIN  combination  systems  that  takes  Into  consideration  the  probability 
associated  with  the  rule  as  well  as  the  evidence.  These  probabilities  are 
called  edge  merits,  and  are  used  to  propagate  the  values  of  evidence  through 
rules  as  well  as  heuristic  In  rule  conflict  resolution.  The  edge  merit  for 
an  AND  combination  Is  defined  as: 


EL Hi 

P(E) 


The  eage  merit  for  an  OR  combination  Is  defined  as: 


IrPtHl 

l-P(E) 

Then*  when  the  rule  Is  fired*  the. AND  function  returns  the  minimum  of  the 
probabilities  of  Its  arguments  and  the  OR  function  returns  the  maximum  of  its 
arguments. 

Edge  Merits  can  be  Incorporated  Into  a  data  fusion  scheme  that  uses  any  of 
the  above  methods. 

3.6  Endorsements  and  Endorsers 

Cohen  and  Grlnberg  (Cohen  &  Grlnberg*  1983)  developed  a  theory  of  heuristic 
reasoning  about  uncertainty  which  Is  symbolic  (non-numerlcal )  and  uses  some 
of  the  concepts  from  Doyle's  Truth  Maintenance  System.  They  propose  a  system 
In  which  evidence  for  an  Inference  Is  associated  with  that  Inference  and  Is 
called  an  Endorsement  for  that  Inference  (much  like  Justifications  In  a  Truth 
Maintenance  System).  Thus*  endorsements  are  records  of  the  Inferences  which 
have  taken  place*  and  Endorsers  are  defined  as  the  computations  that  assert 
these  records.  (See  section  on  Nonmonotonic  Reasoning.)  Unfortunately*  no 
further  work  on  Implementing  a  system  based  on  the  theory  has  been 
published.  Nonetheless*  the  theory  Is  Interesting  enough  to  be  briefly 
explained  below. 

Cohen  and  Grlnberg  claim  that  the  numerical  approaches  to  reasoning  under 
uncertainty  restrict  the  amount  of  heuristic  knowledge  about  uncertainty  and 
evidences*  knowledge  that  humans  actually  use.  In  numerical  methods*  a 
number  Is  merely  a  summary  of  the  evidence  which  supports  an  inference*  and 
actual  evidence  Is  left  unaccesslble  by  the  processes  reasoning  about  future 
Inferences.  There  are  two  reasons  why  the  evidence  for  an  Inference  should 
be  accessible.  First*  a  number  cannot  relate  the  type  of  evidence  which 
supports  the  belief.  Some  types  of  evidences  will  have  more  Importances  In 
certain  situations  or  contexts  (l.e.  corroborative  evidence  vs.  contradictory 
evidence)*  and  knowing  what  kind  of  evidence  Is  supporting  an  Inference  can 
only  aid  the  reasoning  process.  The  second  reason  to  make  the  evidence 


accessible  Is  the  belief  of  an  Inference  Is  based  on  the  context  of  Its 
evidence*  which  Includes  the  current  Inference  being  reasoned  about. 
Therefore*  evidence  used  for  one  Inference  Is  In  a  different  context  from 
evidence  used  for  another  Inference*  and  thus  the  belief  In  the  evidence 
Itself  can  be  different  between  the  two  contexts.  This  becomes  most 
Important  when  dealing  with  evidence  which  supports  an  Inference  being  used 
as  evidence  for  a  new  Inference  (l.e.  propagation;  numerical  methods  can 
often  yield  meaningless*  and  Irrelevant  belief  values  through  propagation). 

The  certainty  of  an  Inference  Is  represented  by  the  strongest  endorsement  for 
the  Inference.  Therefore*  an  Inference  supported  by  one  kind  of  endorsement 
(eyewitness  evidence)  would  have  a  higher  certainty  than  If  the  Inference  was 
supported  by  less  preferred  evidence  (circumstantial  evidence).  This  means 
that  knowledge  Is  needed  to  define  and  rank  a  characteristic  (or  primitive) 
set  of  domain  endorsements.  Also*  knowledge  Is  needed  to  heurlstlcally 
propage  endorsements  over  Inferences  (much  like  degrees  of  belief  are 
numerically  propagated  over  Inferences  (much  like  degrees  of  belief  are 
numerically  propagated  over  Inferences  by  combining  functions)*  but  the 
propagaton  must  be  sensitive  to  the  context  of  the  Inference.  Rules  are 
needed  to  propagate  endorsements  over  Inferences*  thus  serving  the  same 
purpose  as  combining  functions*  with  each  domain  of  expertise  having  numerous 
Idiosyncratic  rules  for  covering  special  cases  of  endorsement  propagation. 

The  Endorsement  Theory  Is  a  fresh  look  at  reasoning  with  uncertainty*  but  the 
theory  leaves  many  unanswered  questions.  The  knowledge  and  structure  of  the 
rules  Is  unclear*  as  well  as  the  final  form  of  each  Inference.  These  holes 
will  have  to  be  addressed  before  the  theory  can  be  Implemented  Into  a  working 
system.  One  benefit  of  the  theory  Is  that  by  retaining  the  endorsements  of 
an  Inference*  one  can  discount  the  uncertainty  of  the  evidence  once  the  use 
of  the  Inference  Is  known  (l.e.  discounting  the  uncertainty  of  one  value  when 
It  Is  averaged  with  other  values).  In  this  manner*  a  better  grip  on  the 
propagation  of  uncertainty  Is  maintained*  thus  yielding  a  more  understandable 
belief  of  the  Inferences. 


Traditional  reasoning  systems  based  upon  predicate  logics  are  considered 
monotonic  In  that  the  number  of  statements  known  to  be  true  Is  strictly 
Increasing  over  time.  All  newly  added  statements  and  newly  proven  theorems 
cannot  disprove  any  of  the  previously  known  knowledge-  In  the  system* 
Unfortunately  the  world  Is  not  monotonic;  but  Is  Incomplete*  constantly 
changing*  and  In  order  to  reason  efficiently  over  complex  problems* 
assumptions  (default  reasoning)  must  be  made  about  the  world  which  can 
possibly  be  proven  Incorrectly  later  In  the  reasoning  process.  From  this* 
the  development  of  Non-Monotonl c  Logics  has  begun  [Doyle  and  McDermott*  1979* 
McDermott  and  Doyle*  1980]*  and  though  the  theory  Is  fairly  new*  Its  basic 
concepts  allow  default- reasoning  assumptions  to  be  made  and  retracted  without 
disrupting  the  belief  Integrity  of  the  world  model. 

Default  reasoning  allows  the  Inclusion  of  logical  statements  on  the  order  of 
"If  X  cannot  be  proven  with  you  have  right  now*  then  conclude  Y"  In  the 
problem  solving  process.  Thus  assumptions  can  be  made  and  considered  true 
until  proven  wrong.  When  a  contradiction  Is  found*  backtracking  Is  performed 
to  the  assumption  which  caused  the  contradiction  and  then  that  statement  and 
all  statements  derived  from  It  are  withdrawn  from  the  world  model.  This  Is 
called  dependency-directed  backtracking  [Stallman  and  Sussman*  1977]. 

Non-monotonlc  reasoning  systems  have  two  added  components  over  common 
Inferenclng  systems.  The  first  Is  an  Assunptlon  mechanism  which  creates 
assumptions  based  usually  on  defaults  about  partial  solutions  to  aid  In 
solving  the  problem.  The  second  component  Is  a  truth  maintenance  mechanism 
which  upon  the  discovery  of  a  contradiction  finds  the  faulting  statement  and 
revises  the  truth  of  the  world  model  by  withdrawing  the  statement  and  all  of 
Its  dependents  from  the  world.  Most  of  the  work  In  building  non-monotonlc 
reasoning  systems  has  been  based  on  Doyle's  Truth  Maintenance  Systems  (TMS) 
[Doyle  1979*  a&b]  which  Is  an  Implemented  system  that  supports  non-monotonlc 
reasoning  by  serving  as  a  truth  maintenance  subsystem  available  to  other 
reasoning  programs.  The  system  does  not  generate  new  Inferences  but 
maintains  the  Integrity  and  consistency  across  the  statements  produced  by  the 
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reasoning  program  via  Its  own  dependency-directed  backtracking  mechanism. 
Since  TMS  Is  the  basis  of  most  research  In  non-monotonlc  reasoning*  It  Is 
explained  below. 

In  TMS*  each  statement  Is  called  a  ’node’  and  Is*  during  the  reasoning 
process*  either  believed  to  be  true  (IN)  or  not  believed  to  be  true  (OUT). 

OUT  statements  are  not  believed  because  there  exists  no  reason  for  believing 
them*  or  because  none  of  the  possible  reasons  for  believing  are  currently 
true.  Associated  with  each  node  Is  a  list  of  justifications  which  reflects 
how  the  validity  of  one  node  can  depend  on  the  validity  of  others*  and  of 
which  these  are  two  kinds:  Support  Lists  [SL  (In-nodes)  (out-nodes)]  and 
Conditional  Proof  [CP  (consequent)  (In-node)  (out-node)].  Supports  Lists  are 
the  most  common  and  Its  node  Is  IN  If  all  of  the  In-nodes  are  IN  and  all  of 
the  out-nodes  are  OUT.  If  there  are  no  In-nodes  nor  out-nodes  then  the 
statement  Is  considered  a  'premise.'  Conditional : Proofs  are  hypothetical 
arguments  which  hold  derived  contradictions  within  the  world. 

As  an  example*  suppose  we  are  Inferenclng  as  to  what  type  of  animal  I VAN  Is. 
We  will  start  out  with: 


Nl. 

Animal  (IVAN) 

*  Shark 

IN 

[SL  (  )  (N2)] 

N 2. 

Animal (IVAN) 

<>  Shark 

OUT 

N3 . 

Animal (IVAN) 

<>  Dog 

OUT 

The  system  assumes  that  IVAN  Is  a  shark  (node  1  Is  IN)  since  there  Is  no 
reason  to  believe  It  Isn't.  From  these  the  system  could  Infer  that  since 
Ivan  Is  a  shark*  then  he  can  also  swim: 


N4.  IVAN  can  swim 


IN  [SL  (Nl)  (  )] 


To  show  the  malntenence  system  In  action*  assume  that  new  Information  Is 
Introduced  by  statements  N25  and  N32  (from  else  where  In  the  system)  and 
found  to  contradict: 


N5.  IVAN  has  fur 


IN 


[SL  (N25*  N32)  (  )] 


The  Inference  system  realizes  a  contradiction  In  that  sharks  do  not  have  fur 
and  so  the  dependency  directed  backtracking  Is  triggered.  In  searching  back, 
the  system  realizes  that  N1  Is  an  assumption  and  could  be  causing  the 
contradiction.  First*  the  contradiction  Is  marked: 

N6.  CONTRADICTION  IN  CSL  (Nl,  N5)  (  )] 

N7.  NO-GOOD  Nl  IN  [CP  N6  (Nl,  N5)  (  )] 

The  NO-GOOD  node  marks  the  proof  that  If  Nl  IVAN  Is  a  shark  and  N5  IV W  has 
fur  then  N6  we  have  a  contradiction.  Without  N7*  we  would  never  know  that  we 
tried  Nl  If  we  ever  have  to  backtrack  again.  Next*  we  update  the  rest  of  the 
world  model  and  select  a  new  assumption: 


Nl. 

Animal (IVAN)  * 

Shark 

OUT. 

CSL 

(  ) 

(N2,  N3)3 

N2. 

Animal (IVAN)  <> 

Shark 

IN 

CSL 

(N6) 

(  )] 

N3. 

Animal (IVAN)  * 

Dog 

IN 

CSL 

(N2) 

(N8)  ] 

N4. 

IVAN  can  swim 

OUT 

CSL 

(Nl) 

(  )] 

N5. 

Animal ( IVAN)  <> 

Dog 

OUT 

N6. 

CONTRADICTION 

OUT 

[SL 

(Nl* 

N5)  (  )] 

Now  the  Inference  mechanism  can  proceed  and  derive  new  deductions* 
assumptions  and  contradictions. 

There  are  a  number  of  variations  to  the  TWS  scheme  [London  1978;  Thompson* 
1979;  Ginsberg*  1984]  but  all  are  based  on  the  same  theoretical  concepts  and 
all  offer  considerable  Improvements  over  other  classical  systems.  Since  the 
Inferenclng  and  dependency  directed  backtracking  Is  non-chronologlcal*  the 
support  relationships  rather  than  the  temporal  orderings  determine  the 
recovery  from  an  error  and  thus  the  appropriate  erronlous  assumptions  are 
found  quickly.  Also  the  use  of  Conditional  Proof  (CP)  structures  allows  the 
causes  of  contradictions  to  be  summarized  and  recorded*  thus  mistakes  are 
made  only  once.  These  two  Improvements  provide  an  Increase  In  efficiency 
large  enough  to  offset  the  overhead  of  maintaining  the  Justifications  (though 
overhead  would  depend  on  Implementation.)  Difficulty  In  using  TMS  could 
develop  when  marking  an  assumption*  (l.e.  IVAN  Is  a  shark)  when  all  other 
possibilities  must  be  created  and  accounted  for  In  the  out-list  of  the 
assumption: 


KS 


Nl. 

Animal (IVAN) 

= 

Shark 

IN 

N2. 

Animal (IVAN) 

<> 

Shark 

OUT 

N3 . 

Animal ( IVAN) 

<> 

dog 

OUT 

N4. 

Animal (IVAN) 

<> 

Platypus 

OUT 

[SI  (  )  (N2*  N3...n)3 


Possibly  mechanisms  can  be  created  to  Introduce  alternative  assumptions*  one 
by  one*  as  they  are  needed.  This  would  greatly  reduce  overhead. 

Using  the  Non-monotonlc  Logic  provides  great  advantages  over  classical  logics 
and  Is  an  excellent  augmentation  to  uncertainty  Inferenclng  systems.  To 
date*  no  one  has  yet  published  an  Implementation  of  a  truth  maintenance 
system  complete  with  uncertainty  factors  for  Its  assumptions.  Possibly*  the 
establishment  of  both  (non-monotonlc  and  uncertainty)  theories  will  allow 
development  of  an  Integrated  theory.  For  a  brief  view  of  TMS  see  [Rich* 

1983*  Doyle*  1979a] #  and  see  [Doyle*  1979b]  for  an  extended  view.  For  the 
mathematical  background  on  Non-monotonlc  Logic*  see  [McDermott  and  Doyle* 
1980;  Doyle*  and  McDermott  1979c*]. 


Fuzzy  set  theory  Is  a  well -developed  mathematical  theory  which  has  not  yet 
been  significantly  exploited  In  A. I.  systems.  It  provides  an  augmentation  to 
evidential  reasoning  systems  which  must  reason  with  fuzzy  qualified 
statements  such  as:  "most  extremely  fat  women  are  very  sweaty"#  where  "most"* 
"fat"*  "sweaty"#  "extremely"#  and  "very"  use  fuzzy  qualities  (not  exact 
boundry  between  membership/ non-membership  of  the  quality.) 

The  theory  of  fuzzy  sets  Is  the  development  of  a  body  of  concepts  and 
techniques  for  systematically  dealing  with  Imprecise  boundaries  between 
classes  of  objects.  A  fuzzy  set  Is  a  class  In  which  there  Is  gradual 
progression  from  membership  to  non-membership  and  each  object  In  the  set  has 
a  grade  of  membership  Intermediate  between  1  (full  membership)  and  0 
(non-membership).  Thus  a  conventional  set  Is  a  degenerate  case  of  a  fuzzy 
set  where  only  two  grades  of  membership  are  allowed:  1*  0. 


The  need  ot  a  fuzzy  logic  Is  due  to  the  fuzzyness  of  the  world.  We  often  use 
qualifiers  In  describing  membership  to  a  set  such  as:  very*  quite*  almost* 
slightly.  In  order  for  a  system  to  reason  about  fuzzyness  It  must  first 
represent  It.  Let  U  be  our  universe  of  discourse  (e. g.  set  of  Integers)  and 
A  be  a  fuzzy  subset  of  U*  which  Is  characterized  by  a  membership  function: 

vA  »  U  ->  CO*  1] 

which  associates  with  each  element  u  of  U  a  number  va(u)  In  the  Interval  CO* 
1]*  with  va<u)  representing  the  grade  of  membership  of  u  In  A. 

Example;  Let  the  universe  of  discourse  be  the  set:  U  *  Cl*  2*  3*  4,  5...] 
with  u  Interpreted  as  "small".  A  fuzzy  subset  of  U  labeled  •’very"  may  be 
defined  as: 

very  small  *  (vj_:ui)+  (v2:u2>+  <V3:u3>+  (V4:u4)+  (v5:u5> 

*  1:1  +  1:2  +  0.5:3  +  0.25:4  +  0.05:5 

where  :  Is  a  separator  to  avoid  confusion  and  +  means  union  of  the  elements. 

The  members:  1*  2  of  "small "  have  a  grade  of  1*  while  the  member  3  has  a 
grade  of  0.5  (l.e.  not  as  very  small  as  1  or  2  are*  and  so  on.)  From  here  a 
number  of  theoretical  concepts  can  be  defined  Including  classical  set 
definitions  (l.e.  containment)  set  operations  (l.e.  complement*  union)* 
relationships  and  other  principles.  For  a  more  thorough  account  of  fuzzy  set 
theory  see  CZadeh*  1977]. 

Fuzzy  logic  Is  an  extension  of  fuzzy  set  theory  which  provides  a 
representation  for  fuzzy  quantifiers  and  truth-values*  as  well  as  provides  a 
set  of  translation  and  Inference  rules  which  can  reason  over  the  fuzzy 
representation.  Fuzzyness  Is  represented  through  possibility  distributions 
Tx  *  the  possibility  distribution  of  X)  which  Is  a  heurlstlcally  derived 
measure  of  the  semantic  fuzzyness  of  linguistic  variables.  Let  F  be  a  fuzzy 
subset  of  U  *  (0*  1*  2...)  and  "X  Is  F"  represents  F  ■  Tx»  or: 

"X  Is  F  ->  Tx  »  F" 


An  examples 

Let 

X  Is  small  ->  Tx  =  1:0  +  1:1  +  0.8:2  +  0.6:3  +  0.4:4  +  0.2:5. 

Then 

PossiX  *  0)  *  1 
Poss {X  *  1}  =  1 

I 

I 

Poss {X  =  5J  =  0.2 

Where  PosstX  =  ul  *  vp(u)  Is  the  possibility  that  X  may  take  u  as  a  value. 

So  If  I  say  "there  Is  a  small  number  of  people  left"*  the  possibility  that 
there  Is  only  2  left  Is  PosstX  =  2)  »  vp(2)  =  0.8.  Possibility  Is  different 
from  probability  In  that  probability  Is  a  measure  of  randomness  while 
possibility  Is  a  measure  of  semantic  fuzzyness  or  Imprecision  In  value. 

A  variable  In  fuzzy  logic  Is  considered  a  linguistic  variable  whose  values 
are  represented  as  words  or  sentences  In  a  natural  or  synthetic  language. 

The  value  of  each  variable  defines  a  possibility  distribution  In  the  domain 
of  the  variable.  For  example:  given  the  primary  variable  TRUE,  Its  antonym 
FALSE,  and  a  finite  set  of  modifiers  and  connectives  such  as  ami,  ur,  Dflt, 
very*  more  or. Jess,  extremely,  etc...  the  linguistic  value  of  true  may  be 
generated  and  represented  as: 

True  False 

not  true  not  false 

very  true  very  false 

not  very  true  not  very  false 

more  or  less  true  more  or  less  false 

I 

I 

not  true  and  not  false 

not  very  true  and  not  very  false 

I 

I 

The  linguistic  truth-value  Is  a  composition  of  possibility  distributions  of 
the  primary  variable  and  attached  modifiers.  Translation  rules  provide  a 
means  of  deriving  the  composite  truth-values.  Translation  rules  fall  Into 
the  four  categories  as  explained  below. 


1.  Modification  rule  (not»  very#  more  or  less#  etc.) 


If  X  Is  F  ->  Tx  *  F 
Then  X  Is  mF  ->  Tmx  =  F+ 

where  m  Is  a  modifier  of  F  and  F+  Is  a  modification  on  F. 

Example:  Let  F  -  small#  Tx  *1:0  +  1:1  +  0.8:2  +  0.6:3  +  0.4:4  +  0.2:5. 
Let  m  =  very#  and  F+  =  <F)2  (F  squared) 

Then: 

X  Is  very  small  — >  Tmx  =  (F)2  * 

1:0  +  1:1  +  0.64:2  +  0.36:3  +  0.16:4  +  0.04:5 

2.  Conjunctive#  disjunctive  and  Impl Icatlonal  rules:  Let  F  and  G  be  fuzzy 
subsets  of  U  and  W# 

X  Is  F  -->  Tx  =  F  and  Y  Is  G  -->  Ty  =  G 

a.  X  Is  F  and  Y  Is  G  — >  T(x#yj  *  F  x  G  where: 

v(FxG)*u,w*  *  m*n  *  vF(u),  vq(w)) 

b.  X  Is  F  or  Y  Is  G  — >  T(x,y)  »  F  U  G  (union) 

where: 

F  *  F  x  W  (compliments) 

"g  =  g  x  u 

v(p  u^)^u»w)  *  max  ( vp( u ) # vq( w) ) 

c.  If  X  Is  F  then  Y  Is  G  -->  T(x/y)  *  F  8  G 

where  T(x/y)  Is  the  conditional  possibility  distribution  of 
Y  given  X  and 

V(F8G)(u»v)  *  M1n(l,  (l-vp(u)  +  vG(w)  )  ) 


Example;  Let 


F  *  SMALL  =  lsl  +  0.6:2  +  0.1:3 
G  =  LARGE  =  0.1:1  +  0.6:2  +  1:3 

Then 

X  Is  small  and  Y  Is  large  *  T(x#y) 

=  0.1: (1* 1)  +  0.6: ( If  2)  +  1: (1,3)  +  0.1:(2,1) 

+  0.6: (2,2)  +  0.6: (2,3)  +  0.1: (3,1) 

+  0.1 : (3,2)  +  0.1 : (3,3 ) 

3.  Quantification  rule  (many,  few,  several,  all,  some,  etc...): 
If  U  =  tu,  ...un),  Q  Is  a  quantifier  and 

F  =  v^:u2  +  V2:u2  +  •••  +  vn:un 
X  Is  F  — >  Tx  «  F 

then  "QX  are  F"  (eg.  "several  X's  are  large")  translates  to 


Tcount(F)  *  Q 


where 


^count(F)  - 


N 


IE  V1 

1=1 


Examp  Lb= 

Let 

SEVERAL  =  0:1  +  0.4:2  +  0.6:3  +  1:4  + 

1:5  +  1:6  +  0.6:7  +  0.2:8 
Then  SEVERAL  X's  are  LARGE  =  17' 

H  i/  „ 

*.  i  L  C  vL'#  / 

*  *  '  * 


0:1  +  0.4:2  +  0.6:3  + 
+  1:6  +  0.6:7  +  0.2:8 


where  vlarqeCuj)  Is  the  grade  of  membership  of  the  1th  value  of  X  In  the 
fuzzy  set  LARGE. 


4.  Truth  qualification  rules  Let  t  be  a  fuzzy  truth-value  like  "very 

true"*  "quite  true"*  etc.  Then  "It  Is  t  that  X  Is  F"  Is  expressed  as: 


X  Is  F  Is  t  — >  Tx  =  F+ 

where 

yF+tu)  ■  v-t<vp(u)) 

Example: 

Bob  Is  young  Is  very  true 

where: 

m  =  very  — > 

then 

Tage(Bob)  *  vtrue^^ vyoung^u^» 
where  u  Is  an  element  In  the  Interval 

[0*1003 

assuming 

vyoung*u*  ( 71  J  J 

and 

vtrue*w)  “  W2»  w  Is  an  element  In 

the  Interval  [0*  13 

then 

.  i.i 

Tage(Bob)“  f  1  ■*  (•&)  J 


The  translation  rules  above  can  be  combined  to  provide  the  possibility 
distribution  of  composite  propositions.  From  here*  rules  for  fnferenclng 
over  the  possibility  distributions  can  be  defined.  Zadeh  CZadeh  793 
Identifies  complex  rules  for  projection  and  conjunction  and  combines  them 
Into  a  composite  rule  of  Inference  which  Is  a  generalized  version  of  the 
classical  modus  ponens.  Thus  he  claimed  to  be  able  to  Infer  the  possibility 
distribution  of  Y  from  the  knowledge  of  X's  possibility  distributions  and 
from  the  conditional  possibility  distribution  of  Y  given  X.  It  Is  through 
this  composite  rule  of  Inference  that  Zadeh  claims  will  expand  the 


applicability  of  rule-based  systems*  giving  them  an  Interpol atlve  (Fuzzy) 
capability.  Fuzzy  logic  Is  not  an  alternative  to  the  other  work  In  this 
survey*  but  Is  an  augment Ive  representation  which  can  be  Integrated  with 
non-monotonl c  logic  and  evidential  (uncertainty)  reasoning.  All  of  these 
theories  are  complex  and  Implementing  an  Integrated  system  of  them  may  be 
close  to  Impossible.  Only  after  they  are  Individually  developed  will  that 
question  be  answered.  For  the  best  coverage  of  fuzzy  set  and  logic  theory 
see  CZadeh*  1977*  Zadeh*  19793. 


23 


4.0  PLANNING  AND  CONTROL 

A  plan  Is  a  partially  ordered  net  of  operations*  each  performable  by  a  host 
unit.  Planning  Is  the  process  of  ordering  the  operators  such  that  their 
actions  and  resource  needs  do  not  conflict.  In  a  Multi  source  Information 
Fusion  System  there  can  be  potentially  high  data  rates*  and  thus  sensor 
resources  and  data  processing  controls  needs  to  be  managed  effectively*  with 
the  collection  of  Important  Information  being  optimized  and  Irrelevant  data 
filtered  out.  Decisions  must  be  made  as  to  what  sources  (of  Information)  on 
which  to  concentrate;  what  processes  to  run;  when  and  where  to  run  them;  what 
parameters  to  use;  and  which  data  to  pass  along.  Memory*  bus*  processors  and 
time  must  all  be  allocated.  The  entire  Information  collection  process  must 
be  planned  out  such  to  optimize  the  collection  of  vital  Information.  Though 
Garvey  and  Lowrance  [Garvey  and  Lowrance*  1984]  promote  the  Instigation  of 
planning  within  a  MSIF  system*  current  planning  strategies  have  yet  to  be 
adopted  for  MSIF  designs.  Garvey  has  looked  at  the  use  of  planning  In  vision 
systems  [Garvey*  1976*  Ballard  and  Brown*  1982]  but  his  approach  Is  very  high 
level  (l.e.  searching  for  a  telephone  In  an  office  by  first  planning  to  find 
the  table  and  searching  the  table  top).  Very  little  work  has  been  done  In 
applying  automated  planning  strategies  to  plan  the  control  of  large  MSIF 
systems. 

The  current  state-of-the-art  In  Automated  Planning  uses  a  hierarchy  of  goals 
to  create  a  plan  [Sacerdotl*  1977]*  and  meta-level  knowledge  to  control  the 
plan  creation  [Hayes-Roth  et  al»  1979*  Steflk,  1981].  Given  a  high  level 
goal  with  constraints*  the  goal  Is  expanded  Into  a  partially  ordered  net  of 
(children)  subgoals  and  actions  which  will  achieve  the  (parent)  goal.  Then 
the  subgoals  are  examined  and  compared  to  an  Internal  world  model  to  see  If 
It  Is  already  true.  If  not*  the  net  Is  examined  to  see  If  restructuring  the 
plan  will  make  the  subgoal  true  and  If  not*  the  subgoal  Is  made  true  by 
expanding  It  Into  more  subgoals.  This  process  continues  until  all  goals  are 
true.  Interml tently  the  plan-net  Is  examined  for  conflicts  between  goals 
(such  as  attempting  to  use  the  same  resources  at  the  same  time).  Often 
conflicts  are  re solved  by  reordering  the  plan-net.  Goals  are  kept  In 
parallel  until  a  necessary  ordering  can  be  determined  [Sacerdotl*  1977]*  and 
variables  are  not  arbitrarily  bound  but  have  constraints  (which  describe  the 
value)  placed  on  them  until  a  correct  value  Is  found  [Steflk*  1981  a&b]. 
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The  processes  Involved  In  planning  are  quite  simple;  the  difficulty  Is  In 
placing  knowledge  about  the  host  unit's  (In  MSIF  a  source  system  controller) 
actions  Into  a  planning  system  operator.  The  effects  prerequisites 
constraints  on  variable  bindings  of  when#  how#  and  why  (Intentions)  to  use 
the  action  and  much  more  must  be  acquired  from  all  possible  actions  of  the 
host  unit.  Incomplete  or  Incorrect  knowledge  can  cripple  the  planning 
process. 

At  the  end  of  the  planning  process  the  final  plan  can  progress  through  a 
number  of  optimization  and  Interpretation  processes  [Kempf#  19831  which  will 
transform  the  plan  Into  a  program  for  the  controller  to  follow.  Possibly  the 
controller  will  have  these  processes  built  Into  It  and  will  Intelligently  use 
the  plan  as  a  knowledge  base  which  guides  the  controller's  actions  CBudenske# 
1984]. 


The  duty  of  the  controller  Is  to  manage  the  operation  of  the  sensors.  With 
the  high  flux  of  possible  data  to  be  collected#  It  Is  desirable  to  focus  the 
sensors  on  portions  of  the  external  world  where  pertinent  Information  Is  more 
likely  to  be  extracted.  Not  only  can  the  high  data  rate  be  cut  by  selective 
sensing  but  also  by  selective  processing  of  the  raw  data.  The  controller 
will  receive  a  plan  from  the  planning  module  and  will  follow  this  plan 
allocating  system  resources  managing  sensors  executing  processes  and  passing 
the  resulting  knowledge  on  to  the  data  fusion  module.  Possibly#  the 
controller  will  also  control  the  data  fusion  and  Inferenclng  processes 
through  assignment  of  Initial  probabilities: uncertainties  and  other  parameter 
adjustments.  The  exact  methods  of  planning  and  control  have  yet  to  be  fully 
examined  and  thus  provide  an  excellent  area  for  fruitful  extension  of  the 
state-of-the-art. 


5.0  SUMMARY 


The  research  In  Multi  source  Information  Fusion  systems  is  very  lacking.  The 
majority  of  work  has  been  In  the  area  of  evidential  reasoning  with  the  tasks 
of  planning  and  control  being  Ignored  as  well  as  the  overall  design  of  an 
Integrated  system.  The  problems  In  evidential  reasoning  have  been  approached 
through  different  Inferenclng  mechanisms  and  logical  theories*  all  of  which 
are  still  in  the  early  periods  of  development.  The  integration  of  some  of 
the  mechanisms  and  logical  theories  could  greatly  Increase  the  evidential 
reasoning  capabilities  of  a  MSIF  system*  but  these  mechanisms  and  theories 
must  be  furthered  developed  before  integration  can  occur. 
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THE  ROLE  OF  STRUCTURE  IN  HUMAN  AND  MACHINE  PERCEPTION 


ABSTRACT 


For  some  time*  there  has  been  a  growing  awareness  among  the  Image  Understanding 
(IU)  research  community  that  the  traditional  approaches  were  not  yielding 
satisfactory  results  In  terms  of  desired  performance.  This  Is  strongly  related 
to  the  emphasis  In  traditional  IU  on  segmenting  and  characterizing  distinct 
regions  or  edge  segments.  Relationships  between  regions  have  not  received  much 
attention. 

In  contrast*  researchers  In  human  perceptual  processes  have  long  been  aware  of 
the  Importance  of  the  use  of  relationships  between  regions*  leading  to  grouping 
of  regions.  This  capability  to  form  groups  of  highly  related  regions  Is  a 
fundamental  (low-level)  form  of  structuring  the  Information  In  an  Image. 

Further  work  has  shown  that  humans  make  extensive  use  of  the  symmetry 
properties  of  Image  configurations  In  building  up  Internal  symbolic 
representations  of  the  perceived  Images.  It  has  become  apparent  that  this 
capability  Is  not  a  trivial  one*  and  that  human  facility  In  working  with 
structural  groupings  and  symmetric  relations  develops  only  In  the  latter  stages 
of  childhood. 

These  factors  provide  strong  argument  that  one  of  the  major  needs  of  Image 
understanding  systems  now  Is  a  robust*  generic  method  for  representing  and 
processing  both  group-oriented  and  symmetry-oriented  structural  properties  of 
Images.  This  paper  Illustrates  a  method  for  representing  the  low-level 
(grouping)  structure  of  segmented  Images.  The  structuring  process  Is 
thoroughly  based  on  an  Implementation  of  the  factors  which  a  dominant  role  In 
hunan  perceptual  grouping  processes:  similarity*  proximity*  containment*  and 
similar  directionality. 

The  Implementation  scheme  has  been  applied  to  both  natural  (FLIR)  and 
artificial  (Bongard)  Images.  The  resulting  Hlerarchal  Relational  Structures 
(HRS)  provide  an  organizing  schema  for  grouping  related  regions  for  further 
processing.  The  HRS  further  enables  a  potentially  simpler  form  of 
representation  for  higher-order  relationships  (such  as  symmetry)*  and  provides 
a  succinct  structure  to  which  meaning  or  object/ feature  Identification 
processes  can  be  applied. 
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1.0  INIR00UCTI0N:  THE  NATURE  OF  THE  PROBLEM 


The  ultimate  goal  of  machine  visual  perception  Is  to  create  a  complete* 
multlaspect  symbolic  representation  of  the  "world"  corresponding  to  the 
Image.  This  capability  would  then  lead  to  further  capabilities*  e.g. *  an 
ability  to  "Image"  the  same  scenario  from  other  perspectives*  or  by 
manupulatlng  the  Internal  symbolic  representation*  to  "Image"  potential 
changes  In  the  scene*  as  shewn  In  Figure  1.1.  These  desired  capabilities 
for  machine  perception  correspond  to  known  human  capabilities  In  perception 
and  Imagery. 

This  paper  presents  an  approach  to  Improving  machine  perception 
capabllltes*  or  machine  Image  understanding  ( IU) *  by  drawing  substantially 
from  known  aspects  of  human  perceptual  processes.  First*  though*  It  Is 
worthwhile  to  briefly  note  the  status  of  machine  IU  performance. 

Currently*  IU  systems  can  produce  "segmented  images"  from  raw  Image  data. 
CSee«e.g.*  Ballard  and  Brown*  1982;  Marr*  19823.  These  segmented  Images 
are  usually  one  of  two  sorts:  those  which  show  all  the  lines  or  edges 
which  can  be  found  In  the  original  Image*  and  those  which  show  all 
contiguous  regions.  Methods  which  combine  these  approaches  are  also 
available.  CE.g.,  Mllgram,  19793  For  the  purpose  of  demonstrating  the 
concept*  we  will  use  region- segmented  Images,  but  the  problems  and  methods 
will  be  applicable  to  either  type  of  segmented  image. 

Once  a  region-segmented  image  Is  obtained*  IU  systems  tend  to  characterize 
each  of  the  segmented  regions  as  much  as  possible  [Panda*  1978;  Ballard  and 
Brown*  19823,  Typically  this  Involves  obtaining  values  for  region  area  and 
extent*  describing  the  shape*  color*  or  texture  of  the  region*  and  In  other 
ways  characterizing  the  attributes  or  content  of  the  region.  Recent  work 
has  emphasized  methods  which  would  yield  depth  or  orientation 
characteristics  of  the  region  [Ballard  and  Brown,  1981;  Barrow  and 
Tenenbaum*  19813. 

Each  of  the  three  major  stages  discussed  above;  the  original  Image*  the 
segmented  Image,  and  the  more  symbolically  represented  2-D  or  2- 1/2-0 
surface  knowledge  (from  the  characterized  regions)  CMarr,  19763*  comprises 


a  major  form  of  knowledge  representation  about  the  original  scene  (from  which 
the  Initial  Image  was  taken)  CBarrow  and  Tenenbaum*  19813.  Minimally*  two  more 
levels  of  knowledge  representation  are  desired.  The  characterized  surfaces  or 
regions  described  above  need  to  be  represented  In  a  different  Internal  form 
which  clearly  makes  expllclty  the  3-0  nature  of  objects*  features*  and  the 
background  or  terrain.  In  many  cases*  this  will  Involve  a  form  of  geometric 
3-0  object-centered  representation*  where  each  object  can  be  uniquely 
represented  with  full  symbolic  description  of  Its  shape.  (Excellent  work  has 
been  done  along  these  lines  by  Brooks  et  al*  and  by  other  groups  of 
Investigators).  [Brooks*  Greiner*  and  Blnford*  1979;  Brooks*  1981;  Brooks* 
19833.  Other  representation  schemas  may  be  used  for  background*  terrain* 
and/or  extended  terrain  features  [Sedgwick*  19833. 

This  does  not  complete  the  process  of  Image  understanding.  While  a  3-0 
geometric  representation  of  a  perceived  scene  Is  necessary*  it  must  be 
supplemented  greatly  by  knowledge  about  the  Identification  and  meaning  of 
each  object*  feature*  and  aspect  of  terrain.  At  this  level*  objects  would 
be  represented  In  terms  of  function  rather  than  form.  Functional ly-rel ated 
object  classes  would  exist*  as  would  (potentially)  schemas  for  events  or 
object  configurations  [Tsotos.  1984;  Havens  and  Mackworth,  19833. 

Thus*  the  traditional  approach  to  conceptually  designing  an  Image 
understanding  system  uses  the  idea  of  multiple  knowledge  representation 
levels,  hierarchically  arranged  (as  shown  In  Fig.  1.2).  The  major  levels 
of  knowledge  representation  are  (roughly):  original  Image*  segmented 
Image*  symbolic  representation  and  characterization  of  the  segmented  image* 
(Including  depth  and  orlentaton  information)*  3-D  geometric  form-based 
representation*  and  a  topmost  "meaning"  or  "Interpretation"  level.  Each  of 
these  levels*  of  course*  can  obtain  several  different  sub-levels. 

There  are  several  major  bottlenecks  which  lie  between  the  current 
state-of-the-art  In  machine  IU  and  the  desired  capabilities  for  machine 
IU.  These  bottlenecks  appear  to  be  strongly  associated  with  major 
transitions  In  the  levels  of  representation  (shown  In  Fig.  1.2). 
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Figure  1.1  Complete*  multiaspect*  symbolic  representation  of  a  perceived 
Image  can  lead  to  the  ability  to  create  perform  "Imagery" 
associated  with  manipulating  the  symbolic  representation. 


KNOWLEDGE 

REPRESENTATION 


REPRESENTATION 


LEYEL 


IMPLEMENTATION 


j  1 

!  Scene  Representation! 

i  j 

Frames,  Semantic 

Nets 

/ 

\ 

/ 

• 

i  * 

i 

;  3-D  Geometrical,  Form- 
i  Based  Object  Representations 

Object  Primitives 
(Generalized  Cones, 
Spheres,  Surfaces), 
Terrain  Maps 

/ 

_ ^ _ 

I 

|  Segmented  Surfaces  and 
|  Intrinsic  Characteristics  j 

U2h3. _ _ _ .! 

7\ 

. . 

i  Segmented  Image 

j  AND 

I  2-D  Features 

- TFT - 


Input  Image 


\  Sensor  / 

V  ' 


Statistical.  Supface 

SIZE,  SHAPE,  DISTANCE 
TO  CENTER,  ORIENTATION 
CURVATURE,  2 \  D 
INFORMATION 


Pixel  &  Statistical 
Image  file,  feature 

VECTOR  FILE, 
SILHOUETTES,  REGION 
EDGES j  GRADIENTS _ 

Pixel  Image  File 


Figure  1.2  Traditional  machine  IU  systems  are  strongly  hierarchical.  Each 
hierarchical  level  corresponds  to  a  major  representation 
formalism  for  knowledge  associated  with  the  Image. 


There  has  been*  In  recent  years*  an  enormous  amount  of  work  aimed  at 
facilitating  the  transition  between  these  representation  levels. 

Interesting  and  useful  approaches  have  been  suggested*  each  with  greater  or 
lessor  degrees  of  applicability  to  generic  IU  systems  [Brooks*  Greiner*  and 
Blnford*  1979;  Brooks*  1981*  1983;  Blnford*  1982  (and  references  contained 
therein);  Oshlma*  1983]. 

In  most  recent  years*  effort  has  focused  on  Improved  characterization  of 
depth  or  otlentatlon  which  can  be  obtained  from  many  sources  (shape  from 
texture  [Kander*  1979;  Stevens*  1981;  Beck*  Prazalny*  and  Rosenfeld*  19833* 
shape  from  shading  [Barrow  and  Tenenbaum*  1981;  Smith*  19833*  and  other 
approaches  [Stevens*  1981;  Haber*  19833.  However*  an  awareness  Is  emerging 
that  these  techniques*  no  matter  how  refined  and  sophisticated  they  become* 
will  In  and  of  themselves  be  Insufficient  to  enable  Inter-level 
transitions#  particularly  the  crucial  transition  from  symbolic  2-D  to  3-D 
representation  [Wltkln  and  Tenenbaum*  1983;  Lowe  and  Blnford*  1981*  Lowe* 
19843. 

A  major  theme  of  this  paper  (and  the  theme  of  some  recent  efforts  by  other 
Investigators)  Is  that  the  previous  approaches  place  a  heavy  reliance  on 
extracting  characteristics  (content)  of  regions  In  an  Image.  This  paper 
defines  an  approach  to  Image  understanding  which  Incorporate  more  fully  the 
relationships  between  the  regions*  and  defines  a  higher-level  concept: 
that  of  structure  In  the  image.  The  "structure"  which  will  be  the  subject 
of  this  paper  may  be  preliminarily  defined  as  "the  Interrelation  of  parts 
as  dominated  by  the  general  character  of  the  whole"  [Webster's  Third  New 
International  Dictionary]. 

The  goal  of  this  paper  Is  to  examine  some  aspects  of  the  role  of  structure 
In  human  Image  understanding*  and  to  extract  from  our  knowledge  of  human 
visual  processes  certain  capabilities  which  could  potentially  play  a  strong 
role  In  machine  IU.  It  also  shows  how  these  capabilities  could  Influence 
the  overall  performance  of  machine  IU*  and  Illustrates  the  proposed  use  of 
structure  In  machine  visual  perception  for  both  artificial  and  real  Images. 
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2.0  THE  ROLE  OF  STRUCTURE  IN  HUMAN  VISUAL  PERCEPTION. 


A  premise  for  this  section  Is  that  humans  perceive  and  describe  objects  and 
scenes  In  terms  of  four  major  aspects:  content#  context#  structure#  and 
meaning.  This  simplified  premise  Is  offered  primarily  as  an  organizing 
schemata  so  that  progress  and  capabilities  of  machine  Image  understanding 
can  be  compared  with  human  abilities. 

Briefly  each  of  these  terms  means  the  following: 

o  Content  -  Description  of  an  object  (or  segmented  region  In  a  scene)  In 
terms  of  Its  attributes#  e.g.#  shape#  texture  color#  and  size. 

o  Context  -  Description  of  the  relationships  between  an  object  or 
segmented  region)  and  neighboring  objects  or  regions.  Common 
relationships  to  consider  are  the  so  called  "Gestalt"  relationships: 
proximity#  similarity*  containment  (or  enclosure)*  and  directionality. 

o  Structure  -  Description  of  an  object  (or  configuration  of  objects  or 
regions)  In  terms  of  the  organization  which  describes  how,  overall#  the 
Interrelations  of  the  parts  contribute  to  perception  of  the  whole. 

o  Meaning  -  Naming,  Interpretation#  and/or  connotation  of  a  perceived 
object  or  region,  or  "configuration"  of  objects  or  regions.  Meaning 
can  be  related  to  the  specificity  of  naming,  or  to  the  connotated 
properties  of  the  named  object. 

A  substantial  body  research  has  shown  that  humans  work  towards  and  create 
an  Internal  symbolic  representation  of  objects  or  scenes.  This  Internal 
representation  contains  content#  context#  structure#  and  meaning  for 
perceived  and  Imaged  scenes. 

Although  It  Is  by  no  means  clear  that  humans  operate  with  the  same 
hierarchical  representation  structure  as  has  been  proposed  for  machine  IU 
systems#  It  Is  well  known  that  humans  have  the  capability  to  symbolically 
represent  and  Interpret  different  levels  or  forms  of  visually-related 
knowledge: 


1.  Representation  of  2-0  charactersltlcs  of  a  perceived  object  or 
scene.  This  Is  most  obvious  In  the  case  of  abstract  line  drawings 
or  figures  where  no  three-dimensional  Information  Is  Intended. 
[Vernon,  1953;  Kohler,  1947;  Koffka,  19353 


2.  Internal  representation  of  the  three-dimensional  nature  of  a 
perceived  object  or  scene.  Our  ability  to  mentally  manipulate  such 
Internal  three-dimensional  models  has  been  amply  demonstrated,  most 
notably  by  the  experiments  of  Shepard  [Shepard,  19713. 


3.  Representation  of  classes  of  objects  and  schemas  of  events. 
Hunan  facility  for  organizing  objects  and  events  Is  well  known,  and 
has  formed  a  basis  for  significant  research  In  artificial 
Intelligence  and  cognitive  science  [See,  e.g.,  Minsky,  19753. 


These  human  capabilities  evolve  over  the  course  of  childhood.  Work  by 
Blnet  and  Simon  showed  that  the  evolution  of  scene  understanding  abilities 
In  children  developed  with  age  [Blnet,  19163.  At  three  years  of  age, 
children  could  enumerate  the  objects  In  a  scene.  By  the  age  of  seven,  they 
could  describe  objects.  Between  the  ages  of  seven  and  fifteen,  they 
developed  the  ability  to  describe  events  and  the  the  relationships  between 
objects  and  persons.  Generally,  by  the  age  of  eleven,  they  were  able  to 
Interpret  the  picture  as  a  whole  [Blnet,  1916;  Vernon,  19533. 


There  is  some  similarity  between  the  evolution  of  abilities  In  children  and 
the  process  by  which  adults  Identify  a  figure.  Adults  first  have  a  vague 
"feeling  of  something",  followed  by  a  vague  Impression  of  some  Indefinite 
object.  This  Is  followed  by  the.  "generic  object  stage",  at  which  certain 
parts  of  the  object  stand  out  more  clearly.  The  next  stage  (the  "specific 
object  stage")  Is  one  In  which  the  observer  perceives  an  organization  of 
the  parts  In  the  object  or  figure,  while  the  background  fades  out.  In  the 
last  stage,  naming  of  the  object  occurs  [Vernon,  19533. 
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These  stages*  and  the  evolution  of  perceptual  abilities  described  above* 
could  be  approximately  described  as  a  procession  of  capabilities 
corresponding  to  the  four  aspects  of  perception  given  earlier:  content* 
context*  structure*  and  meaning. 

Early  work  by  researchers  In  perception  showed  that  the  tendency  to  group 
objects  or  regions  based  on  laws  of  Pragnanz  (similarity*  proximity* 
containment*  and  directionality)  played  an  extremely  Important  role  In 
human  perception  CKoffka*  1935;  Kohler*  1947;  Beck*  1966*  1972;  Olson  & 
Attreave*  1970;  Rock  &  Brosgole*  1964].  These  studies  were  later  advanced 
to  show  that  humans  organized  perception  so  as  to  create  an  object  or 
configuration  based  representation  that  yielded  maximal  symmetry  under 
group-theoretl c  symmetry  operators  [Garner*  1970; :Palmer*  1983  (and 
references  contained  therein)]. 

The  perception  of  structure  In  abstract  figure  drawings  Is  an  ability  which 
increases  with  age  In  children.  At  the  age  of  three  and  one  half  to  four 
years*  children  can  recognize  and  reproduce  the  difference  between  open  and 
closed  figures*  and  from  the  age  of  four  they  begin  to  recognize  the 
difference  In  rectilinear  and  curved  figures,  and  to  differentiate  among 
such  figure  types  as  squares  and  triangles.  At  this  age*  they  also  begin 
to  be  able  to  understand  object  relations  [Piaget  and  Inhelder,  1948] . 

Young  children  tend  to  copy  exact  details  of  a  complex  line  drawing, 
juxtaposed  without  any  Idea  of  the  pattern  as  a  whole.  They  have  no 
concept  of  the  relationships  among  the  details  or  between  details  and 
overall  structure.  By  the  age  of  nine  to  ten  years*  children  are  able  to 
Identify  the  main  outlines  of  structure  In  an  abstract  line  drawing.  It  Is 
Interesting  that  naming  and  verbal  analysis  of  structure  appear  very 
Involved  In  the  child’s  ability  to  extract  structural  components.  By  the 
age  of  eleven  to  twelve  years*  children  are  able  to  perceive  main  outlines 
of  structure*  supporting  or  subsidiary  Interrelations*  and  are  able  to 
Integrate  details  In  a  manner  corresponding  to  the  original  Image  [Vernon* 


To  what  extent  Is  structure  an  Important  factor  In  human  visual 
perception?  Many  Indications  Imply  that  structure  plays  a  very  Important 
role.  During  this  discussion*  It  will  be  necessary  to  distinguish  between 
two  levels  of  structure*  will  be  referred  to  as  "low-level"  and 
"high-level”  structure.  Low-level  structure  Is  the  organization  imposed  on 
a  figure  or  scene  by  grouping  related  objects  or  regions  together  according 
to  the  Laws  of  Pragnanz  [Kohler*  1947].  High-level  structure  will  be  a 
reworking  of  the  groupings  found  by  low-level  structure  as  to  perceive  the 
maximal  symmetry  of  a  configuration  [Palmer*  19833.  This  symmetry  may  be 
either  translational*  regarding  a  repeated  pattern  or  figure*  or  It  may 
Involve  rotation  or  reflection  about  a  point  or  axis. 

There  Is  some  evidence  that  the  role  of  structure  Is  to  aid  In  the  human 
capability  to  form  Internal  symbolic  representatons  of  Images.  This  stored 
Internal  symbolic  representation  Is  what  would  allow  us  to  recreate  scenes 
once  they  have  been  viewed*  or  to  create  new  or  similar  scenes  [See*  e.g. ; 
Shepard*  1978].  An  Illustration  of  this  Is  that  humans  tend  to  draw  a 
simplified  version  of  a  complex*  abstract  line  drawing  which  they  are  asked 
to  reproduce  from  memory.  The  line  drawing  produced  Is  not  only  simpler 
than  the  original.  It  also  has  a  higher  degree  of  symmetry  than  the 
original  drawing  [Vernon,  1953;  Shepard,  19783. 

Recent  research  has  shown  that  the  right  hemisphere  of  the  human  brain  Is 
highly  specialized  (In  right-handed  persons)  to  handle  a  wide  variety  of 
spatial  relational  processing.  Damage  to  this  hemisphere  can  result  In 
severely  Impaired  capabilities  to  Interpret  and  understand  Imagery. 

Overall*  research  In  localization  of  brain  functioning  supports  the  view 
that  relationships  between  regions,  and  Information  which  can  be  drawn  from 
these  relationships  (structure)  Is  fundamental  to  human  visual  perception 
[Klmura  and  Durford,  1974;  Levy  et  al*  19833. 
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3.0  THE  ROLE  OF  STRUCTURE  IN  MACHINE  VISUAL  PERCEPTION 


As  stated  In  the  Introduction  section,  the  goal  of  machine  visual 
perception  Is  to  construct  a  complete,  accurate,  multi-aspect  symbolic 
representaton  of  the  "world"  perceived  through  the  Imaging  sensor.  This 
would  enable  an  Intelligent  machine  to  perform  further  operations  (which 
might  be  somewhat  akin  to  Imagery).  In  order  to  be  able  to  construct  a 
useful  Internal  symbolic  representation,  the  machine  would  have  to  have 
Image  understanding  capabilities  similar  to  those  described  earlier  for 
humans.  In  the  context  of  machine  Image  understanding  these  are: 

o  Content  -  Segmentation  of  the  Input  Image  Into  appropriate  regions, 
and  characterization  of  these  regions  In  terms  of  size,  shape,  color, 
texture,  and  other  useful  attributes. 

o  Context  -  Internal  representation  of  the  simple  Interrelationships 
between  regions.  Important  relations  to  take  Into  account  would  Include 
similarity,  proximity,  containment  (or  enclosure),  and  directionality. 

o  Structure  -  Two  levels  of  structure  are  appropriate  even  at  the  2-D 
representation  level.  These  are: 

-  Low-level  structure  -  Grouping  "related  regions"  so  that 
they  may  symbolically  be  considered  as  forming  parts  of  a 
"whole". 

-  High-level  structure  -  Determine  the  group-theoretl c 
operators  which  may  describe  a  configuration  or  pattern 
expressed  In  single  regions  or  groups  of  regions.  This  may 
Involve  Identifying  points  or  axes  of  symmetry*  or 
translational  Invariance  for  repeated  forms.  This  structure 
must  be  robust  enough  to  account  for  effects  Induced  by 
perspective. 
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o  Meaning  -  Recognize*  classify*  and  Identify  Important  objects  and 
features.  Associate  contextual  Information  with  certain  object  classes 
(e.g. *  capabilities  of  or  uses  for  certain  objects).  Interpret  events*  or 
"meaningful"  configurations  of  objects  or  features. 

These  four  desired  capabilities  of  Image  understanding  sy stems  are  geared 
towards  enabling  the  system  to  contruct  an  Internal  symbolic  representation 
of  the  world  corresponding  to  the  perceived  Image.  It  Is  still  likely  that 
at  least  three  major  levels  of  representation  will  be  necessary.  These 
three  levels  would  be: 

o  2-D  Symbolic  -  Symbolic  representation  of  the  segmented  regions  and  their 
attributes  (content),  their  relationships  (context),  and  both  low-  and 
high-level  structures  describing  the  configurations  of  the  regions. 

o  3-D  Form-Based  -  Representation  of  the  probable  three-dimensional  forms 
which  yield  the  Image  obtained  by  the  sensor.  This  can  include 
object-centered  representations  for  segmented  objects*  and  terrain-based 
Information  for  representation  of  the  background  or  terrain. 

o  Object/ Feature  Classes  -  Representation  of  all  Identifying  and 
connotatlve  Information  associated  with  the  perceived  objects,  features* 
and  terrain. 

In  comparison  with  the  desired  capabilities  of  machine  visual  perception 
systems*  existing  systems  have  exceedingly  limited  capabilities.  Current 
machine  vision  systems  focus  on  object  Identification  where  the  domain  of 
objects  under  consideration  Is  relatively  small  LBInford*  1982;  Rosenfeld. 
1983].  These  systems  rely  to  a  great  extent  on  object  attributes,  such  as 
size*  shape*  Intensity  or  color*  texture,  and  velocity  (for  real  time 
vision  systems).  Some  systems  make  use  of  relationships  between  different 
regions  or  edges  of  an  object.  In  these  cases*  the  type  of  relational 
Information  extracted  Is  very  limited*  and  does  not  Inlcude  the  full  range 
of  Information  that  comprises  the  domain  of  the  Laws  of:Pragnanz.  The 
relationships  considered  also  tend  to  be  highly  specific  to  the  object 
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domain*  and  lack  the  generality  of  human  perceptual  groupings  [Brooks* 

1981*  1983;  Oshlma,  1983;  Shlral*  1978],  No  active  systems  take  Into 
account  the  full  nature  of  structure  In  constructing  Internal  symbolic 
representations  of  the  perceived  Image,  although  recent  work  by  Lowe 
represents  a  significant  step  In  this  direction  [Lowe  and  Blnford*  1981; 
Lowe,  1984].  Earlier  work  In  visual  analogic  relations  Is  not  strong 
enough  to  form  a  substantial  base  for  Image  understanding  [Evans*  1968]. 

An  alternative  organization  for  a  machine  Image  understanding  system  Is 
shown  In  Fig.  3.1.  While  this  proposed  system  design  maintains  the 
fundamentally  hierarchical  approach  of  early  designs  for  IU  systems  [Barrow 
and  Tenenbaum,  1981],  It  departs  radically  from  earlier  systems  through 
Inclusion  of  context  ( rel atl onshlps),  and  structure  at  the  crucial  2-D  (or 
2-1/ 2D)  and  3-D  symbolic  levels.  In  this  sense,  there  Is  somewhat  of  an 
analogy  to  the  lateralization  of  visual  processes  In  the  human  brain. 

In  the  2-1/ 2-0  level,  content  Information  leads  to  context,  from  which 
low-level  structure  can  be  gained.  Structure  provides  groupings  of  regions 
which  facilitate  object/ feature  characterization  and  Identification.  Using 
both  2-1/2-D  content  and  structure  Information,  a  fuller  Interpretation  of 
the  3-D  characteristics  of  an  object  or  feature  Is  possible.  At  the  3-D 
level  also,  the  contextual  knowledge  Is  facilitated  by  both  2-1/2-D  context 
and  structure.  Finally,  the  2-1/2-D  structure  leads  Into  3-D  structure, 
which  will  Include  full  3-D  symmetry  understanding. 


Scene  Understanding 


Semantic 

Representation 

Level 


i 


t 

Object/Terrain  Classification 
and  Identification 

High-Level  Structure 


o  3-D  Structure 


o  Symmetry 


Pixel  Array 
.  Representation  Uv«l 
Figure  3.1 


Segmented  Image 

I 

Image 

Conceptual  design  (In  terms  of  representation  levels)  for  a 
machine  IU  system  which  would  Incorporate  use  of  relationships 
between  segmented  regions*  and  Image  structure  based  on  the 
relationships  between  regions. 


4.0  CAPABILITIES  ENABLED  THROUGH  CONSIDERATION  OF  STRUCTURE  IN  IMAGES 

The  structure  of  regions  In  an  Image  plays  an  Important  role  In  enabling 
perceptual  processes  and  Image  understanding.  By  representing  the 
structure  of  segmented  regions*  as  well  as  the  attributes  of  and 
relationships  between  these  regions*  a  new  form  of  Information  Is  made 
available  to  the  knowledge  bases  and  Inference  engines  which  would  operate 
on  the  Image.  For  the  remainder  of  this  paper,  the  emphasis  will  be  on  the 
representation  of  low-level  (grouping)  structure  for  Images*  and  on  the 
knowledge  which  can  be  gained  fran  this  structural  representation  through 
1 nference. 

In  brief*  the  mechanism  for  evolving  a  representation  of  low-level  (Gestalt 
grouping)  structure  Is  as  follows:  Each  segmented  region  In  an  Image  Is 
symbolically  represented  by  a  "node",  which  will  have  appended  to  It 
content  and  context  Information.  A  parameterized  form  of  the  Gestalt  "Laws 
of  Pragnanz"  Is  used  to  determine  the  relationships  between  nodes 
(contextual  Information).  (These  relationships  are  detailed  In  Table 
4.1.)  This  allows  the  most  "closely-related"  nodes  to  be  Identified  and 
grouped  together.  A.  "cluster  node"  symbolically  represents  this  grouping 
of  closely-related  nodes.  Content  and  context  Informaclon  for  the  new 
cluster  node  Is  calculated.  The  process  Is  repeated*  creating  higher 
levels  of  cluster  nodes,  until  the  entire  Image  (or  segmented  object)  Is 
represented  by  only  a  few  cluster  nodes.  Decomposing  the  hierarchical 
cluster-node  structure  allows  access  to  the  components  of  each  group  of 
nodes.  The  result  of  this  process  Is  that  regions  which  would  be  grouped 
together  by  our  human  perceptual  processes  are  similarly  grouped  together 
by  the  structural  process  (referred  to  as  Hierarchical  Region  Structure 
(HRS)). 


An  example  Is  shown  In  Fig.  4.1.  Fig.  4.1(a)  shows  a  Forward-Looking 
InfraRed  (FLIR)  Image  of  a  road.  We  would  naturally  group  together  the  two 
semi-parallel  strips  together  as  a  structural  unit*  or  "configuration". 
Moreover*  we  would  naturally  consider  the  left  and  right  (bright)  sides 
bordering  the  road  each  to  be  "units"  (In  a  symbolic  sense)  despite  the 
fact  that  there  are  three  regions 
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original  FLIR  image  of  a  road  which  will  be  used  to  demonstrate 
ication  of  the  Hierarchical  Region  Structure  method. 


Figure  4.1  (b)  A  simplified  segmented  image  of  the  FUR  road  image  shown  in  Figure 
4.1  (a).  Some  manual  resegmentation  has  been  done  on  this  segmented 
image. 


(c)  The  Hierarchical  Region  Structure  of  the  segmented  road  regions 


(d)  The  hatched  road  regions  correspond  to  the  hatched  lower-level 
circles  in  Figure  4.1  (c).  They  are  symbolically  represented 
by  a  single  node  at  a  higher  level  in  the  HRS  shown  in  Figure 


corresponding  to  the  left-hand  road  border*  and  two  regions  on  the  right. 
Current  Image  processing  algorithms  can  segment  out  these  regions  (to 
greater  or  lesser  degrees  of  satisfaction*  depending  on  the  methods  used)* 
and  characterize  each  of  the  segmented  regions.  However*  a  mechanism  to 
relate  the  segmented  road  regions  together  Into  a  single  structural  whole 
Is  not  a  part  of  generic  1U  technology.  In  this  sense*  the  grouping 
offered  by  the  HRS  approach  represents  a  strong  development  In  IU 
capabilities. 


Fig.  4.1  (b)  shows  a  segmented  Image  based  on  Fig.  4.1  (a).  Fig.  4.1(c) 
shows  the  part  of  the  Hierarchical  Region  Structure  which  relates  to  the 
road  area.  The  hatched  nodes  at  the  bottom  of  Fig.  4.1(c)  correspond  to 
specific  regions  In  the  segmented  Image*  shown  hatched  In  Fig.  4.1(d).  The 
three  left-side  border  road  regions  are  grouped  together  (using  high-value 
relations  of  similarity  *  proximity*  and  similar  direction)*  as  are  the  two 
right-side  border  regions.  These  form  the  two  cluster  nodes  which  can  be 
seen  at  the  second  level  from  the  bottom  of  Fig.  4.1(c).  These  two  nodes 
are  grouped  together  on  the  basis  of  their  similarity  and  similar 
directionality*  and  form  a  new  cluster  node  (third  level  from  bottom). 

This  node  joins  with  a  cluster  node  representing  the  Interior  of  the  region 
to  form  an  overall  road  node  (one  level  down  from  the  top).  Similar 
processes  would  group  together  the  regions  on  the  right  and  left  sides  of 
the  region. 

Using  this  hierarchical  structure*  to  which  content  and  context  Information 
would  be  appended*  an  Inference  engine  could  search  among  the  top  layers  of 
the  structure  for  a  cluster  node  which  has  characteristics  and  Internal 
relationships  (context)  corresponding  to  the  known  content  and  context  of  a 
road.  A  strong  advantage  of  this  approach  Is  that  at  this  point,  all  of 
the  road  regions  would  be  symbolically  linked  together  and  could  be  treated 
as  a  unit*  thus  enabling  further  high  level  processing. 
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Figs.  4.2  -  4.4  further  Illustrate  the  capabilities  offered  by  a 
representation  of  low-level  structure  In  an  Image*  as  well  as  the 
limitations  Inherent  In  this  approach.  These  figures  are  all  based  on 
Figs.  88*  89  and  90  In  Bongard's  one  hundred  ^Problems  for  the  Recognition 
Program"  CBongard,  19703,  Each  of  the  original  problems  consisted  of  two 
sets  of  six  small  figures  each.  .The  goal  In  each  case  was  to  determine 
what  unique  characteristic  described  one  set  of  figures  which 
differentiated  It  from  the  other  set  of  figures  In  the  same  problem.  For 
succinctness*  I  have  used  only  three  of  the  figures  of  each  of  the  sets* 
Instead  of  the  original  six.  Each  of  the  original  figures  used  for 
discussion  here  Is  shown  In  the  left-hand  column*  and  a  hierarchical 
structural  representation  for  It  Is  shown  on  the  right. 

In  the  structural  representation*  each  node  which  Is  a  termination  of  the 
downward-pointing  tree-like  structure  represents  one  of  the  white  or  black 
ellipses  In  the  original  figure.  (The  notations  of  "W"  for  white  and  "B" 
for  black  are  added  underneath  the  terminal  nodes  of  the  hierarchical 
structure  In  order  to  facilitate  comparison  with  the  original.) 

The  figures  In  the  first  set  of  Fig.  4.2  each  have  three  ellipses,  those  In 
the  second  set  of  the  same  figure  each  have  five  ellipses.  In  this  case* 
the  addition  of  an  hlerarchal  structure  which  describes  the  structural 
groupings  of  each  of  the  figures  does  little  to  help.  Differentiation 
between  the  two  sets  can  be  based  on  content  alone.  Such  differentiation 
Is  well  within  the  province  of  traditional  Image  understanding  systems, 
which  could  extract  the  content  of  each  figure*  and  Infer  the  nature  of  the 
set  definitions:  three  or  five  ellipses  per  set*  respectively. 

Fig.  4.3  presents  a  slightly  more  complex  situation.  In  this  case,  the 
content  of  each  figure  Is  not  sufficient  to  fully  differentiate  the  sets. 
Seme  figures  have  three  ellipses*  others  five*  others  as  much  as  fifteen. 
The  key  feature  here  Is  the  grouping  of  the  ellipses  together.  This  Is 
shown  vividly  In  the  accompanying  hierarchical  structures.  Each  of  the 
structures  for  the  first  set  has  three  major 


)blem  (Bongard,  Problem  88)  in  which  structure  is  not  necessary  to  complete  the 
:he  first  set  of  three  figures  (enclosed  in  boxes),  including  HRS  representations 
those  three  figures;  (Set  2),  the  second  set  of  three  figures,  including  HRS 
:he  second  set. 


A  visual  analogy  problem  (Bongard,  Problem  89),  in  which  a  low-level  structural  representation  is 
necessary  for  a  machine  to  solve  the  analogy. 


"cluster  nodes"  (In  an  expanded  use  of  the  term,  since  sometimes  a  "cluster 
node"  will  correspond  to  a  single  node  of  the  original  Image).  Each  of  the 
structures  In  the  second  set  has  five  cluster  nodes.  In  each  case,  the 
structures  are  built  using  heuristics  and  processes  based  on  the  Gestalt 
Laws  of;Pragnanz.  For  simple  figures  such  as  these,  the  resulting 
structures  are  very  strongly  determined,  with  little  ambiguity  In  terms  of 
the  groups  which  will  be  formed.  The  Information  produced  by  this 
Gestalt-like  grouping  or  structuring  process  presents  a  major  step  beyond 
that  provided  by  traditional  IU  systems. 

Despite  the  clear  advantage  that  such  structural  groupings  offer.  It  Is 
Important  to  be  clear  on  the  limitations  of  such  Information.  It  Is 
necessary  to  understand  that  the  Information  offered  by  the  low-level 
structural  process  Is  devoid  of  both  meaning  and  representation  of 
synmetry.  These  limitations  are  Illustrated  by  Fig.  4.4.  Here,  although 
the  hierarchical  structure  clearly  groups  together  the  adjacent  white 
ellipses  before  clustering  them  with  the  black  ellipses.  It  Is  by  no  means 
clear  that  this  Is  sufficient  to  distinguish  between  the  two  sets.  The 
distinguishing  criterion  in  this  case  (three  groups  of  white  ellipses  In 
the  first  set,  four  In  the  other),  would  have  to  be  made  at  a  higher 
Inferential  level  than  that  of  the  previous  figure.  Thus,  while  low-level 
structuring  offers  a  substantial  body  of  Information  In  a  form  which 
enables  rapid  processing.  It  Is  only  another  step  In  developing  mature  IU 
systems;  It  Is  not  by  Itself  sufficient  to  solve  all  IU  problems. 
Nevertheless,  the  advantages  offered  by  the  use  of  low-level  structure  In 
IU  systems  Is  so  strong  that  this  area  warrants  active  attention. 
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5.0  CONCLUSION:  ASSESSMENT  MD  IMPLICATIONS 


For  some  time*  there  has  been  a  growing  awareness  among  the  Image 
understanding  resear  rr  unity  that  the  traditional  approaches  were  not 
yielding  satisfactory  results  In  terms  of  desired  performance.  These 
traditional  approaches  have  been  heavily  weighted  towards  obtaining  the  t 

characteristics  of  segmented  regions.  Even  the  more  recent  work*  leading 
potentially  to  realization  of  Marr* s  2-1/ 2-D  primal  sketch#  has  simply 
focused  on  obtaining  depth  or  orientation  cues  about  regions#  which  Is  an 
elaboration  of  the  characterization  process. 

In  contrast  to  this  approach#  researchers  In  human  perceptual  processes 
have  long  been  aware  of  the  Importance  of  the  use  of  relationships  between 
regions#  leading  to  grouping  of  regions.  This  capability  to  form  groups  of 
highly  related  regions  Is  a  fundamental  (low-level)  form  of  structuring  the 
Information  In  an  Image.  Further  work  showed  that  humans  make  extensive 
use  of  the  symmetry  properties  of  Image  configurations  In  building  up 
Internal  symbolic  representations  of  the  perceived  Images.  It  has  become 
apparent  that  this  capability  Is  not  a  trivial  one#  and  that  human  ability 
In  working  with  structural  groupings  and  symmetric  relations  develops  only 
In  the  latter  stages  of  childhood. 

These  factors  provide  strong  argument  that  one  of  the  major  needs  of  Image 
understanding  systems  now  Is  a  robust#  generic  method  for  representing  and 
processing  both  group-oriented  and  symmetry-oriented  structural  properties 
of  Images.  This  paper  Illustrates  a  method  for  representing  the  low-level 
(grouping)  structure  of  segmented  images.  The  structuring  process  Is 
thoroughly  based  on  an  Implementation  of  the  factors  which  a  dominant  role 
In  human  perceptual  grouping  processes:  similarity#  proximity# 
containment,  and  similar  directionality. 

The  Implementation  scheme  has  been  applied  to  both  natural  (FLIR)  and 
artificial  (Bongard)  Images.  The  resulting  Hlerarchal  Relational 
Structures  provide  an  organizing  schema  for  grouping  related  regions  for 
further  processing.  \The  HRS  further  enables  a  potentially  simpler  form 


of  representation  for  higher-order  relationships  (such  as  symmetry)*  and 
provides  a  succinct  structure  to  which  meaning  or  object/ feature 
Identification  processes  can  be  applied. 

While  this  paper  has  brought  up  the  need  for  representing  higher-order 
(symmetry-based)  structure*  It  has  not  attempted  to  demonstrate  a  form  or 
mechanism  for  building  this  representation.  The  factors  which  might  be 
most  fruitful  yields  form  this  work  are  the  realizations  that  first*  Image 
structure  representations  store  a  valuable  form  of  Image-based 
Information.  Second*  Investigation  Into  human  visual  processes  can 
continue  to  yield  valuable  Insights  and  Ideas  for  generic* 
machine-implemented  Image  understanding  systems.  Thirdly*  while  structure 
may  be  rightly  regarded  as  a  valuable  component  In  Image  description*  It  Is 
not  to  be  confused  with  Interpretation  or  understanding  of  the  Image.  In 
this  perspective*  the  use  of  structure  In  machine  Image  understanding  needs 
to  be  regarded  as  one  among  the  many  possible  elements  which  can  ultimately 
enable  full  Image  understanding. 
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ABSTRACT 


This  paper  describes  a  knowledge-based  Image  processing  system  for  the 
segmentation  of  outdoor  scenes.  The  system  consists  of  four  main  units  which 
perform  the  tasks  of:  goal  determination*  preprocessing*  segmentation*  and 
region  evaluation.  The  system  uses  Information  on  the  mission  goals*  sensor 
characteristics*  and  data  measured  from  the  scene*  as  well  as  knowledge  about 
the  performance  of  the  Individual  image  processing  operators. 

The  hierarchical  Image  understanding  system  described  here  has  as  Its  primary 
goal  to  segment  visual  and  Infrared  Imagery  of  outdoor  scenes.  It  performs 
Image  or  region  resegmentation  and/or  reprocessing  by  Intelligently  deciding 
what  Image  processing  operations  to  use*  Measure"  the  effects  on  the  current 
Image  and  make  a  decision  as  to  what  the  next  operation  should  be.  The  system 
has  been  Implemented  In  a  Symbolics  3600  as  a  collection  of  production  rules 
acting  on  a  blackboard-type  scene  knowledge  representation  called  Archival 
Scene  Model.  Representative  results  are  described  In  the  paper. 
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1.0 


INTRODUCTION 


One  of  the  fundamental  challenges  In  computer  vision  Is  obtaining  useful 
segmentation.  In  region  based  segmentation*  the  goal  Is  to  outline  regions 
which  accurately  correspond  to  actual  structures  In  the  scene.  The  quality  of 
the  segmentation  Is  crucial  to  effective  machine  Image  understanding.  Without 
the  capability  for  good  segnentatlon  of  physically  meaningful  regions*  even  the 
most  Intelligent  processing  cannot  achieve  satisfactory  scene  Interpretation. 

The  partitioning  of  a  scene  Into  regions  based  on  additional  knowledge*  such  as 
range  or  other  ancillary  knowledge*  can  yield  very  meaningful  scene 
Interpretation  results  C13.  Traditional  state-of-the-art  Image  segmentors  use 
some  background  adaptive  technique  and  threshold  In  order  to  meet  their 
requirements.  However*  such  techniques  are  "blind"  In  their  adaptation  and 
could  be  severely  affected  by  object  size  or  sensor  characteristics 
1 nstabll Ity . 

In  the  past*  the  most  useful  segnentatlon  results  have  been  achieved  using 
model  driven  techniques  which  were  tuned  to  work  for  specific  situations. 

These  segnentatlon  methods  work  well  for  very  focused  applications  such  as 
finding  bright  targets  In  low  clutter  scenes.  The  performance  of  these 
segmentors  goes  down  rapidly*  however*  when  they  have  to  deal  with  a  wider 
range  of  imagery.  There  ,a re  many  applications  where  computer  vision  systems 
have  to  function  over  a  wide  range  of  situations.  For  example*  the  computer 
vision  system  for  an  autonomous  land  vehicle  must  function  In  a  variety  of 
situations  which  will  be  affected  by  different  sensors*  terrains*  weather 
conditions,  and  even  the  time  of  day.  There  are  currently  no  segmentation 
methods  that  work  well  over  a  wide  variety  of  situations. 

A  way  to  Improve  segnentatlon  results  over  a  wide  range  of  scenarios  Is  to  use 
AI  techniques  to  bring  more  knowledge  to  bear  on  the  problem.  Using  AI 
techniques*  such  as  knowledge  on  the  terrain  or  weather  condltons  can  guide  and 
control  the  segmentation  process.  This  knowledge  can  come  frcm  other  sensors, 
the  processing  results  from  previous  frames,  or  even  from  pre-mission  training. 
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Various  knowledge  driven  Image  processing  techniques  have  been  studied  by 
many.  Duane  et.  al.  [1985]  use  knowledge  driven  production  rules  to  evaluate 
the  region  segmentation  of  an  Image  C2J.  This  evaluation  is  used  to  group 
smaller  regions  Into  larger  ones.  This  evaluation  can  also  resegment  a  region 
with  new  parameters.  The  segmentation  Is  performed  by  a  single  routine  that  Is 
data  driven  using  no  other  knowledge.  On  the  other  extreme  Nazlf  and  Levine 
[1984]  use  production  rules  to  control  all  aspects  of  the  segmentation  process 
[3].  Rules  control  the  analysis  and  groupings  of  1 1 nes.  and  regions  as  well  as 
the  scheduling  of  different  segmentation  tasks. 

Our  knowledge  based  segmentation  approach  makes  use  of  an  In  house  existing 
library  of  Image  segmentors.  The  knowledge  based  control  chooses  segmentors 
from  this  library  of  operators.  It  uses  external  knowledge,  and  knowledge 
extracted  from  the  Image,  to  choose  the  best  operator  and  parameter  values  for 
a  given  situation. 
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APPROACH 


To  use  knowledge  effectively  *  the  algorithm  segmentation  process  Is  divided 
Into  Independent  knowledge  driven  algorithm  modules.  Each  of  the  algorithm 
modules  performs  a  specific  step  In  the  processing  flow.  To  make  a  given  unit 
knowledge  driven*  any  knowledge  that  could  possibly  aid  In  the  task  of  the 
module  must  be  Identified  and  quantified.  It  Is  not  Important  where  this 
knowledge  comes  from#  just  that  it  can  be  known  and  can  be  useful  In  performing 
the  module's  task.  For  example#  an  algorithm  module  to  do  noise  cleaning  could 
use  knowledge  about  the  overall  contrast  of  the  Image  to  determine  which  noise 
cleaning  operators  to  use.  This  knowledge  could  come  from  actual  measurements 
of  the  Image  or  It  could  be  derived  from  other  knowledge  such  as  the  type  of 

sensor  used#  the  weather  conditions#  and  the  time  of  day. 

All  of  the  Information  known  about  a  scene  Is  held  In  a  central  data  based 
called  the  global  knowledge  base  (GKB).  The  GKB  handles  all  of  the  knowledge 
transfer  between  different  modules  In  the  system.  Each  module  takes  knowledge 
It  needs  from  the  GKB  and  returns  any  new  knowledge  generated  to  the  GKB. 

Once  the  pertinent  knowledge  has  been  Identified,  the  operation  of  each  module 
Is  designed  to  be  controlled  In  terms  of  this  knowledge.  Production  rules  have 
proved  to  be  a  good  method  for  knowledge  based  control  within  a  module.  A 

typical  rule  Is  made  up  of  an  antlcedent  and  a  consequent.  The  antlcedent 

consists  of  one  or  more  tests  of  Information  In  the  GKB.  If  all  of  the  tests 
In  the  antlcedent  are  true,  then  the  consequent  Is  executed.  The  consequent 
consists  of  one  or  more  actions  that  can  either  change  Information  In  the  GKB 
or  execute  an  Image  processing  function.  Using  production  rules  makes  the 
control  of  the  segmentation  processing  knowledge  driven#  but  also  makes  It 
flexible  and  modifiable.  Production  rules  can  be  added  or  deleted  as  new  types 
of  knowledge  or  new  Image  processing  routines  are  added  to  the  system. 


3.0 


SYSTEM  DESCRIPTION 


A  prototype  knowledge  driven  segmentation  system  has  been  developed  In  the 
System  and  Research  Center's  Image  Research  Laboratory.  The  system  has  shown 
good  results  on  a  number  of  Images  from  a  wide  range  of  scenarios.  An  overview 
description  of  the  Implemented  system  Is  presented  In  the  remainder  of  this 
section.  Figure  1  shows  a  block  diagram  of  the  prototype  knowledge  based 
segmentation  system.  The  processing  Is  divided  among  four  modules  which  do: 
goal  determination#  preprocessing,  segmentation,  and  region  evaluation.  Each 
of  these  modules  has  a  separate  knowledge  base  that  contains  the  production 
rules  specific  to  the  module,  and  has  access  to  the  GKB  for  scene  information. 

The  goal  determination  module  determines  the  segnentatlon  goals  from  the  system 
goals.  The  system  goals  are  usually  based  on  the  specific  mission,  but  they 
could  also  conceivably  change  during  a  mission.  An  example  goal  determination 
rule  from  our  prototype  system  Is: 

RULE  #15:  IF  ( G0AL= T ARGET_R£C0GN IT ION )  THEN  (SHAPE_G0AL=C0NVEX) 

This  rule  suggests  that  If  the  system  goal  Is  to  recognize  targets  then  the 
shapes  sought  will  be  convex. 

The  preprocessing  algorithm  shown  In  Figure  2,  takes  In  the  original  Image  and 
preprocesses  It  to  accomplish  tasks  such  as  noise  cleaning  and  Image 
enhancement.  This  module  uses  knowledge,  such  as  sensor  type,  to  determine  the 
preprocessing  steps.  The  preprocessing  module  can  also  generate  knowledge, 
such  as  the  overall  Image  contrast,  to  be  used  by  other  modules.  A  sample 
preprocessing  rule  from  our  prototype  system  Is: 

RULE  #08:  IF  (H IGH_FREQ_N0ISE )  THEN  RUN(WINDOW_AVERAGE_ROUTINE> 

For  this  rule  It  Is  not  Important  how  It  was  determined  that  the  Image  has  high 
frequency  noise  present.  It  Is  just  Important  that  the  noise  Is  present.  The 
Information  could  have  come  from  Image  measurements  or  from  prior  reasoning 
based  on  the  sensor  type  and  the  time  of  day.  After  preprocessing  comes  the 
actual  segmentaton. 


The  segmentation  module#  shown  In  Figure  3*  takes  In  the  preprocessed  Image  and 
applies  segmentation  operators  from  a  library  of  possible  operators.  Each 
operator  In  the  library  has  strong  points  and  weak  points.  Table  1  shows  the 
suite  of  segmentation  operators  which  are  currently  In  the  system.  It  Is  the 
job  of  the  segmentation  module  to  choose  the  best  operator  based  on  the 
segmentation  goals  and  on  the  current  scene  knowledge  from  the  GKB.  Besides 
doing  the  actual  segmentation,  the  segmentation  module  generates  knowledge  to 
be  put  In  the  GKB  for  use  by  other  modules.  This  Includes  Image  knowledge# 
such  as  the  total  number  of  regions  segmented#  as  well  as  knowledge  about  each 
Individual  region,  such  as  shape  and  contrast.  The  segmentation  module  Is 
sufficiently  general  so  that  It  can  segment  regions  or  entire  scenes.  This 
makes  the  same  module  usable  for  Initial  segmentation  of  full  Images  or 
resegmentation  of  Individual  regions.  An  example  segmentation  rule  frcm  a 
prototype  system  Is: 

RULE  #47:  IF  (SHAPE_GOAL=CCNVEX  and  C0NTRAST<L0W)  THEN  (RUN  TBL) 


This  rule  recommends  the  segmentation  operator  TBL  when  the  contrast  of  the 
Image  Is  low  and  the  goal  Is  to  find  convex  objects.  After  the  Image  Is 
segmented  Into  regions#  the  regions  are  passed  one  at  a  time  to  the  region 
evaluation  module. 


SEGMENTOR 

TECHNIQUE 

APPLICATION 

Texture  Boundary 
Locator  (TBL) 

Texture  Based 

Low  Contrast  Boundaries 

Prototype  Similarity 
Transform  (PST) 

Region  Based 

Low  Texture  Regions 

M70  Digital 
Gradient  (MDG) 

Edge  Based 

High  Contrast  Boundaries 

TABLE  1.  Segmentation  Techniques  Used  By -The  Knowledge  Based 
Segmentation  System. 
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Figure  2.  Preprocessing  module 
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Figure  3.  Segmentation  module 


The  region  evaluation  module*  shown  In  Figure  4,  determines  what  processing 
should  happen  next  for  each  region  found  by  the  segmentation  module.  The 
region  evaluation  module  recommends  that  a  region  be:  accepted*  resegmented* 
or  Ignored.  This  recommendation  Is  based  on  knowledge  about  the  specific 
region  and  on  the  segmentation  goals.  The  segmentation  module*  working 
together  with  the  region  evaluation  module*  produces  the  best  possible 
segmentation  for  the  Image  based  on  the  current  segmentation  goals.  An  example 
region  evaluation  rule  Is: 

RULE  #75:  IF  (G0AL_SIZE= SMALL  and  GOAL_SHAPE= CONVEX 

and  REG I0N_S IZE=L ARGE  and  REG I0N_SHAPE= CON CAVE 
THEN  ( RESEGMENT_REG ION ) 

This  knowledge  driven  architecture  makes  the  best  use  of  all  available 
knowledge  about  the  scene.  It  also  provides  an  approach  for  making  use  of 
Information  across  multiple  sensors.  The  processing  for  each  of  the  sensors 
can  be  designed  to  be  knowledge  driven*  with  all  sensor  processing  paths 
sharing  the  seme  GKB.  This  would  allow  each  sensor's  processing  flow  to  use 
any  knowledge  possible,  even  If  It  comes  from  anothe'r  sensor. 

This  type  of  system  architecture  also  can  be  designed  to  make  use  of 
Information  across  multiple  frames  of  a  scene.  Information  learned  from  the 
processing  of  one  frame  could  be  used  to  help  processing  of  successive  frames. 
For  example*  If  a  convex  object  Is  found  In  one  frame  at  a  certain  size*  this 
Infomatlon  could  be  used  to  look  for  more  convex  objects  of  the  same  size  In 
successive  frames.  This  could  Improve  overall  system  performance  by  giving  the 
system  more  Information  about  what  It  can  expect  to  see  In  a  given  frame. 
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Figure  4.  Region  evaluation  module 


4.0 


EXPERIMENTS  AND  RESULTS 


In  the  following  we  describe  the  results  of  the  Implemented  knowledge  based 
segmentation  system  for  a  sample  experiment  conducted  In  the  Image  Research 
Laboratory  for  ground  scenarios  Imagery. 

Figure  5  shows  a  sample  video  Image  of  a  road  with  obstacles  on  It.  This  Image 
represents  a  road  following  scenario  where  the  goal  Is  to  find  the  road  and 
then  determine  If  there  are  any  obstacles  In  the  road.  Figure  6  shows  the 
result  of  a  texture  based  operator  on  the  road  Image.  Figure  7  shows  the 
results  of  an  edge  based  segnentor  and  Figure  8  shows  the  results  of  a  region 
based  segmentor.  None  of  these  results  are  sufficient  to  Identify  the  road  and 
the  obstacles.  The  knowledge  based  control  breaks  the  path  following  scenario 
Into  two  goals.  The  first  goal  Is  to  find  the  road  by  looking  for  large 
regions  without  looking  for  details.  Using  these  goals  the  segmentation  module 
recommends  running  TBL  on  a  lower  resolution  of  the  Image.  Figure  9  shows  the 
results  of  this  first  pass  of  segnentatlon.  Each  of  the  larger  regions  pass 
the  segmentation  goals  at  this  point  and  are  accepted  by  the  region  evaluation 
module. 

The  second  set  of  segnentatlon  goals  for  this  path  following  scenario 
recommends  to  look  for  small  convex  obstacle  like  regions  within  each  of  the 
larger  regions.  When  the  regions  are  evaluated  with  respect  to  these  new 
goals*  the  region  evaluation  module  recommends  passing  the  larger  regions  back 
to  the  segmentation  module  for  resegmentation.  Figure  10  shows  the 
resegmentation  results  for  the  center  road  region.  Notice  the  obstacles  are 
now  segmented  out.  Figure  11  shows  the  total  Image  segmentation  when  the 
resegmentation  results  are  combined  back  Into  a  full  Image. 
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Figure  7.  MDG  segmentation  of  full  image 


7a 


5.0  SUMMARY 

The  overall  performance*  robustness  and  multi-scenario  operaton  of  a  computer 
vision  system  can  be  Improved  by  making  use  of  all  possible  knowledge  about  a 
scene.  Knowledge  can  be  better  used  by  designing  the  processing  control  to 
make  use  of  all  available  Information.  By  using  production  rules  the  system 
can  be  knowledge  driven  and  also  be  flexible  and  easily  expandable  when  new 
Image  processing  routines  are  added  to  the  system.  A  prototype  experimental 
system  has  been  Implemented  for  knowledge  based  region  segmentation  and 
resegmentation  with  encouraging  results. 

A  much  more  difficult  problem*  than  region  segmentation*  Is  the  classification 
and  labeling  of  the  regions  In  a  scene.  The  approach  we  have  described  In  this 
paper  Is  goal  driven  segmentation  and  therefore  provides  labeling  Information 
about  the  region  If  the  goal  Is  met  satisfactorily.  This  aspect  of  the 
approach  Is  being  explored  as  a  follcw-up  to  the  current  application. 
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ABSTRACT 


This  report  provides  a  brief  summary  of  a  research  project  currently  under  way 
in  the  Honeywell  Systems  and  Research  Center  on  the  use  of  "depth  cues"  in 
general  and  on  optical  flow  In  particular  for  dynamic  scene  analysis.  The  2-D 
optical  flow  fields  are  determined  by  Implementing  a  gradient  technique  which 
relates  the  changes  of  brightness  In  the  Image  sequence  to  the  spatial 
movements  In  the  scene.  The  results  of  the  use  of  this  technique  on  a  sequence 
of  visible  Images  are  given.  Furthermore#  a  multl-sersor  approach  (which  Is 
currently  being  Implemented)  Is  presented  for  the  construction  of  3-D  optical 
field  from  2-D  optical  field  and  range  Information  available  from  a  ranging 
sensor.  This  will  permit  derivation  of  the  complete  3-D  motion  parameters  of 
the  moving  objects  arbitrary  motions  In  the  scene. 


1.0  INTRODUCTION 


This  report  summarizes  the  preliminary  results  of  Internal  Research  and 
Development  (IR&D)  work  on  the  feasibility  of  the  use  of  ’depth  cues’  in 
dynamic  scene  analysis. 

Humans  readily  use  monocular  depth  cues*  such  as  occlusion*  texture  gradient 
and  optical  flow.  Cl*  2]  Isolated  attempts  at  using  some  of  these  depth 
cues  for  computer  vision  and  scene  understanding  have  been  made  by  various 
researchers  C3-193  In  the  last  decade.  However,  no  systematic  effort  on 
studying  the  use  of  these  depth  cues  for  computer  vision  has  been  reported. 

Some  of  the  depth  cues  such  as  occlusion  and  texture  gradient  are  useful  In 
static  scenes.  Others,  such  as  optical  flow,  are  beneficial  In  dynamic 
environments  where  the  position  of  the  viewer  with  respect  to  a  static  scene 
changes,  or  the  viewer  Is  stationary  and  the  positions  of  some  of  the 
objects  In  the  scene  vary  with  respect  to  the  viewer,  or  there  is  a 
combination  of  these  two  cases. 

In  our  study  we  are  considering  the  use  of  occlusion  cues,  texture  gradient, 
and  optical  flow  for  the  extraction  of  three-dimensional  (3D)  Information 
from  a  scene.  In  this  preliminary  report  we  describe  our  approach  for  using 
optical  flow  to  extract  3D  information.  This  will  greatly  aid  the  process 
of  dynamic  scene  analysis,  and  will  facilitate  studies  of  temporally 
changing  object  features.  In  later  reports,  we  will  present  the  results  of 
our  studies  In  using  both  occlusion  and  texture  gradient  cues. 

There  are  several  ways  In  which  the  Hierarchical  Multi  sensor  Image 
Understandlg  (HMIU)  program  can  benefit  from  these  studies.  Depth  cues  In 
general  and  optical  flew  Is  particular  provides  3D  Information  at  the  lower 
level  of  the  hierarchy  which  will  be  crucial  In  single  sensor  Image 
understanding.  Moreover,  In  our  approach  we  plan  to  Initiate  a  multi  sensor 
fusion  process  at  this  low  level  by  combining  2D  optical  flow  (obtained  from 
IR  or  TV)  with  the  range  Information  available  from  a  ranging  sensor  to 
create  3D  optical  flew  of  the  scene.  This  3D  optical  flow  Is  then  used  to 


extract  3D  motion  parameters  of  the  moving  objects.  We  have  to  note  that  by 
using  single  sensors*  only  the  2D  projected  motion  parameters  can  be 
derived.  However*  In  our  mul tl sensor  approach*  the  complete  9  parameters 
needed  for  3D  motion  description  of  moving  objects  can  be  extracted. 

Figure  1  shows  the  various  steps  Involved  In  our  approach.  From  the 
Infrared  or  visual  Images*  2D  optical  flow  will  be  derived  which  will  then 
combine  with  the  range  Information  available  from  the  millimeter  wave  radar 
(or  Ildar)  and  will  lead  to  the  formation  of  the  three-dimensional  (3D) 
optical  flow.  The  nine  motion  parameters  are  obtained  from  each  velocity 
vector.  By  using  the  generalized  Hough  transform  method  the  motion 
parameters  of  the  rigid  body  motions  In  the  scene  can  be  specified.  In  this 
method,  each  velocity  vector  'votes'  for  a  set  of  motion  parameters  and  the 
parameter  values  receiving  the  most  votes  are  selected  to  describe  motion  of 
the  moving  object. 
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2.0  DETERMINING  2-0  OPTICAL  FLOW 


There  are  two  main  approaches  In  the  literature  for  the  determination  of 
optical  flow:  gradient  methods  and  matching  techniques.  In  matching 
techniques*  one  has  to  follow  the  motion  of  prominent  feature  points  from 
frame  to  frame  In  an  Image  sequence.  These  techniques  require  an  Initial 
segnentatlon  of  the  Imagery  and  more  Importantly  they  rely  on  the  finding  of 
the  best  matching  points.  The  gradient  methods*  however*  avoid  the 
correspondence  problem  by  using  the  original  unsegmented  Images  and  making 
use  of  the  gradient  constraint  equation  which  relates  the  changes  In 
brightness  to  the  temporal  changes  In  the  scene.  For  an  object  of  constant 
brightness  U(x»  y»  z)»  the  following  equation  Is  derived 


V  Vx  +  ^  *o 


where 


(1) 


(2) 


*'1/0. 
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We  used  a  gradient  method  [7*133  for  the  determination  of  optical  flow. 

This  technique  Is  an  Iterative  and  robust  method  for  the  determination  of  2D 
optical  flow  V.  Seme  of  the  features  of  this  method  are  as  follows: 

o 


o 


Samples  are  taken  at  discrete  points  In  space  and  time  and 
quantized  In  brightness. 

U,  \l| 

Partial  derivatives  —  *  £2-  and-S  are  estimates  by  averages 

*>y  It 

using  eight  measurements  In  two  Image  frames. 
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The  criteria  that  Is  minimized  Is  the  square  of  the  magnitude 
of  the  gradient  of  the  optical  flew  velocity 
components  ^  t  TJy  : 


The  final  Iterative  relations  for  estimating  V  and  Vv  are  as  the  following: 

x  7 


n+'  —  —  n  _  n  i  j 

%  a  ITjt  -  +  ilj  ITy  +  %3/0*  +  Jl/  +  Uy  )  (S-a) 

%  =  rVy  -  -YiljVy  (S-b) 


where  Is  a  weighting  factor. 


3.0  DETERMINING  3D  OPTICAL  FLOW 


Consider  a  viewing  geometry#  as  depicted  In  the  Figure  2#  defining  the  three 
components  of  3D  optical  flow  as  Wx#  Wy  and  Wz»  and#  using  geometrical 
relations#  the  following  equation  can  be  derived: 


( f-z)Vx  «  Wxf  +  x!wz 

(6-a) 

(f-z)Vy  «  Wy f  +  y*Wz 

(6*b) 

By  using  focal  length  of  the  camera  f;  positional  coordinates  In  the  Image 
x1#  y1;  2D  optical  flow  velocity  components  Vx  and  Vy;  and  depth  Information 
z#  Wz;  the  components  Wx  and  Wy  will  be  computed. 


4.0  DETERMINING  MOTION  PAR  METERS 


The  motion  of  rigid  bodies  In  the  scene  are  described  relative  to  a  frame  of 
reference.  We  assume  that  this  frame  Is  fixed  to  the  viewer.  The  most 
general  type  of  motion  Is  described  by  the  following  relation: 


X  *  XA  +  P 

( 7  •  a ) 

V  =  VA  +  xp 

( 7*  b) 

where  P  =  X-XA 


an  dills  the  rotation  vector  at  a  point. A  on  the  rigid  body.  VA  Is  the 
velocity  component  associated  with  the  translation  velocity  of  the  point  A. 
so  the  rigid  body  motion  can  be  described  by  9  parameters  VA,.ft,  X,  XA. 
Figure  3  shows  the  relations  between  the  viewer  frame  of  reference  and  the 
rigid  body's  frame  of  reference.  For  determlnl ng_fl.*e  first  calculate 
rotational  direction  W  which  Is  defined  as^by  using  three  successive  3D 
optical  flow  measurements  to  obtain  2  acceleration  vectors  a2  W  Is 
found  from  the  following  relations 


W  =  Al  X  a2/<  aj  X  a2  )  (s) 

VT  and  |il|  will  be  determined  from  the  following  relation: 

V  =  VT  +  |jfl|W  X  P  (<3) 

From  each  velocity  vector  the  motion  parameters  will  be  extracted  and  then 
by  using  a  Hough  transform  C20]  like  procedure*  the  motion  parameters  of  the 
solid  objects  will  be  extracted  by  noticing  that  moving  rigid  bodies  will 
have  the  highest  number  of  points  having  the  same  motion  parameters  in  the 
scene.  The  Hough  transform  was  originally  developed  for  the  detection  of 
straight  line  segments  In  Image.  In  this  technique  the  points  In  the  Image 


Reference  Frames. 


are  transformed  Into  lines  In  a  parametric  (1  Ine- Intercept)  space.  Lines  In 
the  si  ope- Intercept  space  corresponding  to  col  Inear  points  will  cross  each 
other  In  one  point.  This  technique  recently  has  been  generalized  for 
detection  of  more  complex  patterns  In  multi-dimensional  spaces. 


5.0  PRELIMINARY  EXPERIMENTAL  RESULTS 


A  sequence  of  12  TV  Images  were  created  from  a  moving  toy  jeep  at  the  rate 
of  30  frames  per  second.  Figures  4  and  5  show  the  first  two  frames  of 
pictures  of  the  moving  jeep.  Using  the  gradient  technique  described 
previously*  the  optical  flow  fields  by  using  different  nunber  of  franes  and 
varying  the  values  ofo^N  and  the  scale  factor  were  obtained.  Figure  6 
shows  the  derived  optical  flow  fields  forQ^=  2.0»  N  *  16  and  scale 
factor  2.  As  can  be  seen  from  Figure  5#  the  outer  boundaries  of  the  jeep 
can  be  easily  Inferred  frcm  the  optical  flow  fields.  Furthermore*  some  of 
the  surface  orientation  properties  of  the  Jeep  have  also  exhibited 
themselves  In  the  optical  flow  field.  This  Is  very  useful  Information  that 
Is  currently  under  Investigation. 


6.0  SUNK  ARY 


In  this  report  a  brief  description  of  a  research  project  currently  under  way 
at  the  Honeywell  SRC/SIP  on  the  use  of  "depth  cues"  In  general  and  optical 
flow  In  particular  for  dynamic  scene  analysis  was  given.  Our  overall 
multi-sensor  approach  for  the  construction  of  3-D  optical  flow  and  the 
extraction  of  3-D  motion  parameters  was  described  and  the  preliminary 
results  of  the  Implementation  of  a  2-D  optical  flow  technique  as  part  of  our 
overall  approach*  on  a  sequence  of  visible  Images  were  presented. 
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BELIEF  MAINTENANCE  FOR  A  FUZZY  REASONING  SYSTEM 


ABSTRACT 


Real  world  problem  solving  often  Involves  (1)  dealing  with  uncertain  and 
Imprecise  knowledge  and  (2)  making  assumptions  which  are  then  verified  or 
denied  by  the  reasoning  process.  Fuzzy  logic  is  presented  as  the  mechanism  for 
dealing  with  measures  of  belief  and  a  maintenance  system  Is  proposed  for 
handling  assumptions  and  the  accrual  of  evidence. 
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BELIEF  MAINTENANCE  FOR  A  FUZZY  REASONING  SYSTEM 
1.0  INTRODUCTION 

Expert  systems  and  the  accompanying  reasoning  systems  are  at  the  forefront  of 
the  artificial  Intelligence  research  that  Is  being  applied  to  computer  vision 
and  Image  under standing  systems  [1*5*7,9*11*16].  The  difficulties  associated 
with  acquiring  and  encoding  expertise  are  a  major  bottleneck  In  the  practical 
application  of  expert  systems.  Expert  knowledge  Is  often  Imprecise  (vague)  or 
uncertain  and  the  rules  used  In  Inferenclng  with  knowledge  In  this  form  must 
model  and  support  measures  of  belief.  Several  methods  based  on  probability 
have  been  presented  and  used  for  dealing  with  uncertainty  [2*6,15],  yet  none  of 
these  methods  deals  well  with  the  Imprecision  or  vagueness  Inherent  In  expert 
knowledge.  Fuzzy  logic*  based  on  fuzzy  set  theory  [19*21*22]*  assigns 
Intervals  of  possible  values  to  a  fact  rather  than  a  single  probabilistic 
value.  Research  Is  being  pursued  In  unifying  the  posslbll Istlc  and 
probabilistic  approaches  [8*13*18]  and  In  the  application  of  fuzzy  logic  to 
expert  systems  [1*8*12*17*20]  and  to  Image  understanding  [14].  Section  II 
explains  the  basic  concepts  of  Inferenclng  with  fuzzy  logic. 

The  knowledge  In  reasoning  systems  Is  also  Incomplete  and  assumptions  must  be 
advanced  or  it  Is  Inconsistent  and  conflicts  must  be  resolved.  Human  reasoning 
makes  use  of  the  assumption  verlf Icatl on/deni al  paradigm.  In  problem  solving* 
the  human  will  assume  certain  facts  In  order  to  drive  the  reasoning  process. 

If  an  assumption  Is  proven  Incorrect*  the  reasoning  based  on  that  assumption  Is 
Ignored  and  the  problem  solving  continues*  possibly  with  another  assumption. 
This  sort  of  non-monotonlc  reasoning  requires  facilities  for  maintaining 
evidence  for  facts  and  resolving  conflicts  [10],  In  [3]»  facts  are  based  on 
endorsements  and  In  [4]  facts  are  considered  true  based  on  a  support  list.  The 
reader  should  Investigate  these  methods;  however*  an  extension  to  the  latter 
method  Is  provided  In  Section  III. 

Section  IV  is  an  example  of  a  belief  maintenance  system  for  fuzzy  logic  applied 
to  the  classification  of  regions  within  a  scene.  The  system  maintains 
dynamically  accrued  facts  and  assumptions  with  measures  of  belief. 
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Fuzzy  set  theory*  Introduced  by  Zadeh  C 193  In  1965*  deals  with  the  notion  of 
Imprecision  In  discrimination  between  sets  of  objects.  Conventional  or  'crisp' 
sets  have  sharp  boundaries  and  objects  either  .belong  to  a  specific  set  or  do 
not  belong  to  that  set.  Fuzzy  sets*  on  the  other  hand*  contain  objects  which 
have  a  degree  of  membership  between  1  (full  membership)  and  0  (nonmembership). 
An  example  may  further  clarify  the  difference.  The  crisp  set  of  BRIGHT  pixels 
may  be  defined  as  all  pixels  where  the  average  pixel  Intensity  Is  above  some 
threshold.  The  fuzzy  set  of  BRIGHT  pixels  assigns  a  degree  of  membership  to 
each  pixel  value*  such  that  pixels  with  a  low  Intensity  have  a  low  degree  of 
membership  In  the  set  of  BRIGHT  pixels  and  vice  versa  for  pixels  of  high 
Intensity. 

Let  S  be  a  set  of  objects  and  F  be  a  fuzzy  subset  of  S*  such  that  for  each  sin 
S*  there  Is  an  associated  degree  of  membership  In  F.  If  S  Is  the  set  of  pixel 
Intensities  (values  0  to  255)  then  BRIGHT  could  be  a  fuzzy  subset  of  S>  where 
each  pixel  Intensity  In  S  has  a  degree  of  membership  In  the  set  BRIGHT.  In  the 
following  example  of  a  four  Interval  set  BRIGHT*  the  degree  of  membership  Is  .3 
for  a  pixel  Intensity  of  120. 

BRIGHT  -  0/ (0-63}  +  .3/(64-127)  +  .7/(128-191)  +  1.0/(192-255) 

If  the  fuzzy  subset  Is  continuous  rather  than  discretized  Into  Intervals*  the 
membership  function  becomes  a  curve  as  In  Figure  1. 
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In  a  rule  based  system  the  predicates  or  facts  as  well  as  the  antecedents  and 
consequents  of  the  rules  are  In  the  form: 

"attribute  of  object  IS/HAS  value". 

If  V  Is  a  variable  for  the  attribute  of  an  object  where  the  domain  of  V  Is  S* 
then  the  proposition  "V  Is  F",  where  F  Is  a  fuzzy  set*  Induces  a  possibility 
distribution*  u*  over  the  set  S»  such  that  u(s)  *  F(s).  F(s)  Is  the  degree  of 
membership  of  s  In  F  and  u(s)  Is  the  possibility  that  V  =  s  given  the  data  "V 
Is  F".  The  differences  between  possibility  values  and  probability  values  is 
heavily  debated  and  not  an  Issue  here*  yet  an  example  may  provide  some  Insight 
Into  the  difference.  In  the  above  fuzzy  set  BRIGHT  we  see  that  pixels  with  an 
Intensity  value  between  128  and  191  are  given  a  .7  degree  of  membership  In 
BRIGHT.  This  can  be  viewed  as  a  70*  possibility  that  a  pixel  Intensity  Is 
BRIGHT*  yet  It  cannot  be  Interpreted  as  .7  probability.  In  probability*  the 
sum  of  all  the  Individual  probabilities  is  1*  while  In  possibility  many  set 
members  may  have  the  value  of  1  (full  membership).  Therefore  the  proposition 
"V  Is  F"  assigns  a  possibility  or  degree  of  membership  based  on  the  fuzzy  set 
F*  to  each  value  V  can  represent. 

Rules  are  represented  In  a  conditional  form  such  as 

"If  V  Is  F  then  W  Is  G" 

where  F  and  G  are  fuzzy  subsets  of  S  and  T  respectively.  This  rule  Induces  a 
conditional  possibility  distribution  u(s*t>  where 

u(s»t)  *  min  (1*  1  -  F(s)  +  G(t)). 

Compound  antecedents  or  consequents  are  dealt  with  by  two  other  compositional 
rules.  The  conjunction  form 

"VI  Is  FI  and  V2  Is  F2  ...  Vn  Is  Fn" 

where  Fj  Is  a  fuzzy  subset  of  Sj»  Induces  a  joint  possibility  distribution 
u(sl*s2*...sn)  where 

u(sl*s2*...sn)  ■  min  CFj(sj)]  for  j«l,n. 
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The  disjunction  form 


"VI  Is  FI  or  V2  Is  F2  ...  Vn  Is  Fn" 

where  FJ  Is  a  fuzzy  subset  of  Sj»  Induces  a  joint  possibility  distribution 
u(sl# s2».. .sn)  where 

u(sl#s2*...sn)  =  max  [Fj(sj)J  for  j«l»n. 

Combinations  of  these  compositional  rules  and  the  conditional  rule  can  be  used 
to  generate  possibility  distributions.  For  example* 

"If  VI  Is  FI  or  V2  Is  F2  then  W  Is  G" 

Induces  the  possibility  distribution  u(sl»s2»t>  where 

u(sl*s2*t)  -  min  (1,  1  -  max  [Fl(sl),F2(s2)3  +  G(t)). 

There  are  also  modification  rules  such  as  "not"  and  "very". 

"V  Is  not  F"  Induces  u’(s)  *  1  -  F(s). 

"V  Is  very  F"  may  Induce  uw(s)  *  CF ( s ) 3  squared. 

Inferenclng  about  fuzzy  knowledge  Is  based  on  modus  ponens.  That  Is*  given  A 
and  the  knowledge  that  A  Implies  B*  then  B  Is  Inferred.  This  Is  a  conjunctive 
of  the  form 

"V  Is  F  and  (If  V  Is  F  then  W  Is  G>" 

where  F  and  G  are  fuzzy  subsets  of  S  and  T  respectively.  This  Induces  the 
joint  possibility  distribution  u(s*t)  where 

u(s»t)  ■  min  CF(s)»  min  Cl#l-F(s)+G(t)]] 

■  min  CF(s)*  1-F(s)+G(t)3. 
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Since  the  possibility  distribution  u(t)  for  "W  Is  G"  Is  what  Is  being  Inferred* 
then  the  maximum  u(s»t)  for  each  t  Is  the  possibility  distribution  for  each 
G(t).  This  leads  to 

u(t)  =  maximum  u(s*t)  over  all  s 

*  max  Cmln  CF<s)»  1-F<s)+G(t)33  over  all  s. 

*  min  CF(s)»  1-F(s)+G(t)]  when  s  Is  known 

The  strength  of  a  fuzzy  set  value  can  be  determined  In  several  ways. 

1.  The  expected  value  may  be  obtained  by  some  method  such  as  mean 
distribution  and  the  strength  of  a  fuzzy  set  Is  the  membership  value  at 
that  expected  value. 

2.  The  strength  could  also  be  the  sum  of  the  membership  values  for  each 
Interval.  This  Is  the  Integral  of  a  continuous  function  over  the  fuzzy 
set. 

3.  A  threshold  membership  value  Is  determined  and  some  measure  of  those 
Intervals  which  exceed  the  threshold  Is  used  as  the  strength  of  the 
fuzzy  set. 
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PROCESSING  REQUEST  FOR  DECISION 


PROCESS  FLOW  DIAGRAM  FOR  A 
FUZZY  BELIEF  MAINTENANCE  SYSTEM 
Figure  2 


Figure  2  gives  an  example  of  a  reasoning  system  for  Image  understanding.  The 
fuzzy  belief  maintenance  system  receives  predicates  from  the  Inference  engine 
and  maintains  consistency  among  beliefs.  Section  IV  gives  an  example  of  this. 

In  reasoning  over  a  set  of  rules  and  facts*  there  may  be  equally  certain  yet 
conflicting  facts*  or  there  may  not  be  enough  current  knowledge  to  confirm  a 
fact.  An  assumption  Is  advanced  and  the  reasoning  process  continues  (as  may 
the  knowledge  acquisition  process).  .At  some  point  In  this  process*  the 
assumption  may  be  confirmed  and  all  knowledge  Inferred  from  the  assumption  is 
updated  accordingly*  or  the  assunptlon  Is  denied  and  the  knowledge  which  Is 
singularly  Inferred  from  the  assumption  Is  removed  from  the  set  of  current 
beliefs. 

Inconsistency  In  the  current  set  of  beliefs  occurs  when  a  belief  or  predicate 
has  two  conflicting  values.  Conflict  resolution  In  this  sense  then  encompasses 
dealing  with  two  confirming  pieces  of  evidence  as  well  as  complementary  pieces 
of  evidence. 

Conflict  resolution  occurs  at  the  Instant  a  conflict  arises.  There  are  four 
meta-rules  for  the  resolution  process. 

1.  If  neither  the  current  belief  of  a  fact  nor  the  new  belief  of  that  fact 
are  based  on  any  assumption,  then  If  the  new  belief  Is  stronger  than 
the  current  belief  (measures  of  strength  discussed  above)*  replace  the 
current  belief  with  the  new  belief  and  propagate  the  new  belief  value 
to  all  the  other  facts  dependent  on  the  fact. 

2.  If  the  current  belief  of  a  fact  Is  based  on  an  assumption  and  the  new 
belief  of  that  fact  Is  not,  then  replace  the  current  belief  with  the 
new  belief  and  propagate  that  value  to  all  the  other  facts  dependent  cn 
the  fact. 

3.  If  the  current  belief  of  a  fact  Is  not  based  on  an  assumption  and  the 
new  belief  of  that  fact  Is*  then  do  nothing. 


4.  If  the  current  belief  and  new  belief  of  a  fact  are  based  on 

assunptlons;  then  If  the  new  belief  Is  "significantly  less  assumption 
based"*  replace  the  current  belief  with  the  new  belief  and  propagate 
the  new  belief  value  to  all  the  other  facts  dependent  on  the  fact. 

In  the  fourth  meta-rule  above;  the  qualifier  "slgnf Icantly  less  assumption 
based”  Is  some  measure  of  Inherited  assumption  which  Is  a  distance  function  in 
the  Inference  network.  Intuitively;  beliefs  based  on  some  distant  assumptions 
may  also  have  Inherited  some  facts;  while  beliefs  based  on  less  distant 
assumptions  are  likely  to  have  Inherited  less  facts.  Practically;  retaining 
the  most  distant  belief  will  mean  less  propagation. 

The  format  for  beliefs  being  maintained  Is: 

(FACT)  (FUZZY  SET)  (BASIS)  (VALUE) 

where  FACT  Is  a  predicate  from  the  knowledge  base  (1e;  REG I0N_ IN TENSITY  Is 
BRIGHT); 

FUZZY  SET  Is  the  fuzzy  set  associated  with  the  fact  (1e;  the  fuzzy  set 
BRIGHT); 

BASIS  Is  the  support  for  the  fact  (this  Includes  a  tag  for  Indicating 
Inheritance  of  assumption;  all  rules  which  Infer  the  FACT;  and  the 
accompanying  facts  which  triggered  the  rules); 

VALUE  Is  the  real  value  of  the  fact  (1e,  128  Is  the  REGION. INTENSITY) . 
If  VALUE  Is  empty  then  the  expected  value  of  the  fuzzy  set  can  be 
computed  and  used  as  the  val ue. 
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In  the  following  example,  the  fuzzy  techniques  and  the  belief  maintenance 
system,  described  above,  are  Integrated  Into  a  region  classification  tool. 
Sensor  Input  to  the  system  Is  provided  by  MMW  and  IR  sensors.  In  addition, 
wind  velocity  In  scene,  sensor  Inclination  (vertical  direction),  aprlorl 
knowledge  (1e,  map),  and  region  knowledge  from  previous  scenes  or  from  other 
regions  In  the  current  scene,  are  Inputs  to  the  system. 

Example  Rule  Base: 

R1  HIGHUINE  OF  SIGHT  FROM  INCLINATION)  ==>>  (SKY) 

R2  (SKY)  *=»  UNLIKELY (ROAD)  and  UNLIKELY(RIVER) 

R3  HIGH (TEXTURE  FROM  MMW)  «>>  (ROAD) 

R4  LOWHEXTURE  FROM  MMW)  and  (WINDY)  *=»  (RIVER) 

R5  HIGH( INTENSITY  FROM  IR)  ==>>  (ROAD) 

R6  L0W(  INTENSITY  FROM  IR)  ==»  LIKELY(RIVER) 

The  quantlflers/qual If lers  preceding  the  predicates,  modify  the  predicates  In 
specified  ways  (see  C18-223).  HIGH  may  shift  all  the  degrees  of  membership  to 
the  right  and  LOW  may  shift  all  the  degrees  to  the  left  as  In  the  following 
example. 

domal n: pixel  values  0-50  51-101  102-152  153-203  204-255 

(average)  INTENSITY  .1  .7  1  .6  .1 

HIGH  INTENSITY  0  0  .1  .7  1 

LOW  INTENSITY  1  .6  .1  0  0 

Thus,  an  IR  Intensity  value  of  100  would  be  a  0  degree  of  membership  In  the 
fuzzy  set  HIGH  INTENSITY,  and  .6  degree  of  membership  In  LOW  INTENSITY.  LIKELY 
Increases  the  degree  of  membership  for  each  Interval,  while  UNLIKELY  decreases 
the  degrees  of  membership.  In  the  following  example,  ROAD  Is  a  fuzzy  set  over 
the  domain  of  depth,  such  that  a  road  Is  less  likely  to  be  classifiable  10  km 
from  a  sensor  than  a  road  2  km  from  a  sensor.  The  following  example 
Increases/decreases  ROAD  by  .2  degrees  of  possibility. 


domalnsMMW  range(km) 

1-3 

3-5 

5-7 

7-9 

9-11 

11- 

(default)  ROAD 

1 

1 

.8 

.6 

.3 

.1 

LIKELY  ROAD 

1 

1 

1 

.8 

.5 

.3 

UNLIKELY  ROAD 

.8 

.8 

.6 

.4 

.1 

0 

The  predicates  and  associated  fuzzy  sets  In  the  example  are  as  follows. 

The  SKY  domain  Is  the  mean  horizon  height  In  an  Image  (512  lines)  such  that  a 
region  between  350  and  400  pixels  has  a  .8  degree  of  possibility  (pixels 
numbered  from  bottom). 


SKY 

degree 


0-101  102-204  205-307  308-409  410-511 

0  .2  .5  .8  1 


The  ROAD  domain  Is  the  depth  of  region  such  that  a  region  at  8  km  has  a  .6 
degree  of  possibility. 


ROAD  - 
degree 


1-3  3-5  5-7  7-9  9-11 

1  1  .8  .6  .3 


The  RIVER  domain  Is  the  depth  of  region  similar  to  ROAD. 


RIVER  * 
degree 


1-3  3-5  5-7  7-9  9-11  11- 

1  .9  .7  .4  .2  .1 


The  WINDY  domain  Is  the  velocity  at  time  of  Image. 


WINDY  « 
degree 


05-5  5-10  10-15  15-20  20-25  25-30  30- 

0  .1  .3  .5  .7  .9  1 


The  LINE  OF  SIGHT  (LOS)  domain  Is  the  Inclination  In  angular  degrees  of 
sensor.  The  possibility  of  regions  belonging  to  a  certain  class  may  be 
Influenced  by  the  line  of  sight  (see  rule  1). 
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The  TEXTURE  domain  Is  based  on  the  mean  Intensity  value  from  MMN  In  a 
granularity  measure  from  1-8*  where  1  Is  low  texture. 

TEXTURE  -123  45  6  7  8 

degree  0  .3  .6  .9  1  .7  .4  .1 

The  INTENSITY  domain  Is  the  mean  pixel  brightness  of  IR. 

INTENSITY  »  0-50  51-101  102-152  153-203  204-25! 

degree  .1  .7  1  .6  .1 

Given  a  value  for  a  fact*  the  fact  and  current  degree  of  membership  for  the 
value  are  added  to  the  list  of  beliefs.  If  no  value  Is  known  but  the  fact  Is 
either  Input  or  Inferred  then  the  fact  Is  added  to  the  list  of  beliefs  and  the 
value  will  be  the  mean  distribution  of  the  fuzzy  set. 

In  the  following  example*  nothing  is  known  at  the  start  of  the  classification* 
but  as  requests  for  more  Information  are  answered  by  the  DASM  knowledge  Is 
accrued  and  a  classification  Is  arrived  at.  The  system  begins  with  the  Initial 
classification  RIVER  (possibly  from  aprlorl  knowledge  of  the  scene). 

1.  (RIVER)  (1  .9  .7  .4  .2  .1)  (A)  () 

Information  Is  requested  by  the  system  to  continue  processing.  The  Information 
may  come  from  sensors*  aprlorl  knowledge*  etc.  LINE  OF  SIGHT  at  a  10  degree 
Incline  Is  Input  from  DASM. 

2.  (LOS)  (.1  .5  .8  1  .8  .5  .1)  ()  (+10») 

Rule  1  Is  triggered*  adding  SKY  to  the  belief  list. 


3.  (SKY)  (.2  .4  .7  .8  .8)  (R1  2)  () 


4.  (ROAD)  (.5  .5  .5  .5  .5  .5)  (R2  3)  O 

Rule  2  Infers  ROAD  above  and  Infers  RIVER  below. 

(RIVER)  (.5  .5  .5  .5  .5  .5)  (R2  3)  () 

Since  the  original  RIVER  was  assumption  based*  It  Is  replaced. 

1.  (RIVER)  (.5  .5  .5  .5  .5  .5)  (R2  3)  () 

The  system  requests  more  Information  from  DASM. 

5.  (TEXTURE)  (0  .3  .6  .9  l  .7  .4  .1)  ()  (6) 

6.  (WINDY)  (0  .1  .3  .5  .7  .91)  ()  (25  mph) 

Rule  3  Is  now  triggered*  Inferring  ROAD  again. 

(ROAD)  (.6  .6  .6  .6  .6  .5)  (R3  5)  () 

Since  the  original  ROAD  and  the  new  ROAD  are  both  fact  based*  the  fuzzy  sets 
are  ORed  (max  of  each  Interval). 

4.  (ROAD)  (.6  .6  .6  .6  .6  .5)  (R2  3,  R3  5)  () 

Rule  4  Infers  RIVER  with  a  value  below  some  threshold  and  the  new  value  for 
RIVER  Is  Ignored.  RIVER  could  have  ben  ORed  with  the  original  value  producing 
the  same  results. 

(RIVER)  (.1  .1  .1  .1  .1  .1)  (R4  5  6)  <) 

Suppose  a  strength  of  .7  Is  needed  to  conclude  a  fact. 

ROAD  has  a  strength  of  .6  and  RIVER  has  a  strength  of  .5  resulting  In  a  request 
from  DASM  for  more  Information  In  order  to  obtain  a  stronger  conclusion.  Here 
the  Intensity  value  from  an  IR  sensor  Is  Input  to  the  system. 


7.  (INTENSITY)  (.1  .7  1  .6  .1)  ()  (45) 


Rules  5  and  6  are  triggered  Inferring  RIVER  and  not  Inferring  ROAD  at  all. 


(ROAD)  (000000)  (R5  7)  0 


ROAD  Is  Ignored  since  the  value  or  strength  is  nil. 


(RIVER)  (1  1  .9  .6  .4  .3)  (R6  7)  () 


The  new  RIVER  Is  ORed  with  the  original  RIVER  resulting  In  a  strong  conclusion 
for  river. 

1.  (RIVER)  (1  1  .9  .6  .5  .5)  (R2  3,  R6  7)  () 

At  this  point*  RIVER  has  a  strength  of  .75  and  the  particular  region  Is 
classified  as  a  river. 


v.v.v.v'v/.' 


-vV.  •»  e.  .■ 


r’v'vl 


5.0 


SUMMARY 


Fuzzy  logic  provides  a  tool  for  belief  measures  In  reasoning  systems  with 
uncertain  and  Imprecise  facts.  The  nonmonotonic  belief  system  proposed  here 
reasons  over  Incomplete  and  Inconsistent  knowledge.  Assumptions  assist  the 
acquisition  of  knowledge  and  can  be  removed  from  the  set  of  currently  believed 
facts  when  the  assumptions  conflict  with  non-assuned  facts.  This  belief  system 
enhances  the  acquisition  and  Inferenclng  of  knowledge  In  a  fuzzy  reasoning 
system. 
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